[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:32 dist-test-slave-r7bh krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:32 dist-test-slave-r7bh krb5kdc[395](info): set up 2 sockets
Jun 19 00:52:32 dist-test-slave-r7bh krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 00:52:33.567503 388 mini_kdc.cc:121] Time spent starting KDC: real 2.061s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 00:52:33.722229 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.154s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 00:52:33.896747 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.174s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 00:52:34.197990 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.301s user 0.003s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/test-user.keytab.
W20260619 00:52:34.932631 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.734s user 0.001s sys 0.003s
Jun 19 00:52:34 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:35.062752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40213
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:35.392052 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:35.392400 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:35.392573 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:35.399915 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:52:35.400080 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:35.400190 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:35.400305 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:52:35.400394 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:52:35.410744 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:35.412106 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:35.413962 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:35.426465 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:35.426517 408 server_base.cc:1034] running on GCE node
W20260619 00:52:35.426414 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:35.426870 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:35.427686 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:35.429189 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:35.430420 408 hybrid_clock.cc:648] HybridClock initialized: now 1781830355430343 us; error 62 us; skew 500 ppm
I20260619 00:52:35.430809 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:35 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:35.438925 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:52:35.440644 408 webserver.cc:460] Webserver started at http://127.0.97.62:37477/ using document root <none> and password file <none>
I20260619 00:52:35.441466 408 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:35.441622 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:35.441962 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:35.445199 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "ba042493ddbd48ab8cdfecb391edeaa4"
format_stamp: "Formatted at 2026-06-19 00:52:35 on dist-test-slave-r7bh"
server_key: "f39bceec61fd1196d46b4603ea52452c"
server_key_iv: "5c78c63651f34e92b93717bff14f496e"
server_key_version: "encryptionkey@0"
I20260619 00:52:35.446015 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba042493ddbd48ab8cdfecb391edeaa4"
format_stamp: "Formatted at 2026-06-19 00:52:35 on dist-test-slave-r7bh"
server_key: "f39bceec61fd1196d46b4603ea52452c"
server_key_iv: "5c78c63651f34e92b93717bff14f496e"
server_key_version: "encryptionkey@0"
I20260619 00:52:35.451558 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:52:35.455268 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:35.456786 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:52:35.457132 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "ba042493ddbd48ab8cdfecb391edeaa4"
format_stamp: "Formatted at 2026-06-19 00:52:35 on dist-test-slave-r7bh"
server_key: "f39bceec61fd1196d46b4603ea52452c"
server_key_iv: "5c78c63651f34e92b93717bff14f496e"
server_key_version: "encryptionkey@0"
I20260619 00:52:35.457361 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:35.473272 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:35.477916 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:35.478274 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:35.498247 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40213
I20260619 00:52:35.498267 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40213 every 8 connection(s)
I20260619 00:52:35.499718 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:52:35.503609 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:35.509769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 408
I20260619 00:52:35.510094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:52:35.512531 388 external_mini_cluster.cc:1428] Setting key d9b1e4c64bd73bbcfe416c29c0786f06
I20260619 00:52:35.515578 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4: Bootstrap starting.
I20260619 00:52:35.520145 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:35.521535 475 log.cc:824] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:35.524951 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4: No bootstrap required, opened a new log
I20260619 00:52:35.537169 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } }
I20260619 00:52:35.537550 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:35.537686 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba042493ddbd48ab8cdfecb391edeaa4, State: Initialized, Role: FOLLOWER
I20260619 00:52:35.538290 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } }
I20260619 00:52:35.538564 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:52:35.538707 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:52:35.538908 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:35.541793 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } }
I20260619 00:52:35.542465 475 leader_election.cc:302] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba042493ddbd48ab8cdfecb391edeaa4; no voters:
I20260619 00:52:35.543140 475 leader_election.cc:288] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:52:35.543251 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:35.544695 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [term 1 LEADER]: Becoming Leader. State: Replica: ba042493ddbd48ab8cdfecb391edeaa4, State: Running, Role: LEADER
I20260619 00:52:35.545253 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } }
I20260619 00:52:35.545554 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:52:35.547993 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba042493ddbd48ab8cdfecb391edeaa4. Latest consensus state: current_term: 1 leader_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } } }
I20260619 00:52:35.548420 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [sys.catalog]: This master's current role is: LEADER
Jun 19 00:52:35 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:35.548418 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba042493ddbd48ab8cdfecb391edeaa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40213 } } }
I20260619 00:52:35.549536 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:35.552264 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:52:35.558475 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:52:35.564200 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:35.517101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.62:40213)
0619 00:52:35.519797 (+ 2696us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:35.519808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:35.519852 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:52:35.531565 (+ 11713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:35.531591 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:35.532050 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:35.532437 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:35.536579 (+ 4142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:35.538302 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:35.540938 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:35.542133 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:35.542283 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:35.550921 (+ 8638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:35.550941 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:35.550970 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:35.551062 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:35.556126 (+ 5064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:35.557358 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:35.557366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:35.557374 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:35.558004 (+ 630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:35.558605 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:35.558612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:35.558614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:35.559368 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:35.562059 (+ 2691us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:35.562430 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 00:52:35.562858 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2477,"thread_start_us":2240,"threads_started":1}
I20260619 00:52:35.571746 487 catalog_manager.cc:1347] Generated new cluster ID: 33f5a5d4afc04af2b1045d4a06434f80
I20260619 00:52:35.571858 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:52:35.585642 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:52:35.587424 487 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:52:35.602247 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba042493ddbd48ab8cdfecb391edeaa4: Generated new TSK 0
I20260619 00:52:35.603207 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:52:35.648746 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:35.994390 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:35.994683 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:35.995095 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:36.003073 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:36.003211 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:36.003402 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:52:36.013674 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:36.015007 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:36.016268 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:36.029922 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:36.029943 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:36.029898 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:36.030819 501 server_base.cc:1034] running on GCE node
I20260619 00:52:36.031455 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:36.032487 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:36.033717 501 hybrid_clock.cc:648] HybridClock initialized: now 1781830356033649 us; error 72 us; skew 500 ppm
I20260619 00:52:36.034099 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:36 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:36.037550 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:52:36.039171 501 webserver.cc:460] Webserver started at http://127.0.97.1:39153/ using document root <none> and password file <none>
I20260619 00:52:36.039880 501 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:36.040023 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:36.040337 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:36.043350 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "170ba22d5b1f4ac28c298ddc56484991"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "ee24c36bc0f3c2dbaedcc1aa8bda09cb"
server_key_iv: "5ac791f3c786e65e833cad02682e51f2"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.044158 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "170ba22d5b1f4ac28c298ddc56484991"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "ee24c36bc0f3c2dbaedcc1aa8bda09cb"
server_key_iv: "5ac791f3c786e65e833cad02682e51f2"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.049912 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 00:52:36.053678 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.055274 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:52:36.055487 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "170ba22d5b1f4ac28c298ddc56484991"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "ee24c36bc0f3c2dbaedcc1aa8bda09cb"
server_key_iv: "5ac791f3c786e65e833cad02682e51f2"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.055732 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:36.071785 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:36.076696 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:36.077033 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:36.077993 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:36.079329 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:36.079438 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.079586 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:36.079660 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.118184 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36231
I20260619 00:52:36.118235 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36231 every 8 connection(s)
I20260619 00:52:36.119513 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:52:36.127202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 501
I20260619 00:52:36.127420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:52:36.128008 388 external_mini_cluster.cc:1428] Setting key c40ee941ead9e8f184f6eb80a1f023e1
Jun 19 00:52:36 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:36.144071 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:36.122049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46331 (local address 127.0.97.62:40213)
0619 00:52:36.122556 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:36.122560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:36.123787 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 00:52:36.125790 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:36.125797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:36.125892 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:36.126019 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:36.127681 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.128488 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.130547 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.130979 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.131092 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:36.136231 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:36.136241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:36.136251 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:36.136301 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:36.140192 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:36.141047 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:36.141057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:36.141065 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:36.141243 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:36.141758 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:36.141767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:36.141775 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:36.142209 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:36.142494 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:36.143222 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 00:52:36.143514 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260619 00:52:36.144608 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:36.122469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40213 (local address 127.0.97.1:46331)
0619 00:52:36.123556 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:36.123604 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:36.125471 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:36.126275 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:36.126284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:36.126833 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:36.127473 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:36.127513 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.128660 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.128664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:36.130392 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:36.130406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.131220 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.131229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:36.132022 (+ 793us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:36.133129 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:36.133163 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:36.135979 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:36.140370 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:36.140379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:36.140396 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:36.140867 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:36.141400 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:36.141404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:36.141406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:36.141615 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:36.142431 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:36.142437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:36.142783 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 00:52:36.143025 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 00:52:36.143429 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":622,"threads_started":1}
I20260619 00:52:36.146322 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40213
I20260619 00:52:36.146739 629 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:52:36.147758 629 heartbeater.cc:511] Master 127.0.97.62:40213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:52:36.150142 440 ts_manager.cc:194] Registered new tserver with Master: 170ba22d5b1f4ac28c298ddc56484991 (127.0.97.1:36231)
I20260619 00:52:36.152410 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:52:36.172672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:36.502741 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:36.503110 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:36.503264 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:36.514307 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:36.514443 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:36.514684 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:52:36.526857 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:36.528213 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:36.529685 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:36.543555 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:36.543577 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:36.543526 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:36.544318 635 server_base.cc:1034] running on GCE node
I20260619 00:52:36.545083 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:36.546247 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:36.547503 635 hybrid_clock.cc:648] HybridClock initialized: now 1781830356547419 us; error 67 us; skew 500 ppm
I20260619 00:52:36.547864 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:36 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:36.551594 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:52:36.553308 635 webserver.cc:460] Webserver started at http://127.0.97.2:37415/ using document root <none> and password file <none>
I20260619 00:52:36.554114 635 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:36.554257 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:36.554581 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:36.557744 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "cf5412dacb3a40cf9d5d16aa51fefba0"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "fa73d364e51bf3745e93d2b39929704a"
server_key_iv: "1067e37284321d181ad5a422e533953a"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.558598 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cf5412dacb3a40cf9d5d16aa51fefba0"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "fa73d364e51bf3745e93d2b39929704a"
server_key_iv: "1067e37284321d181ad5a422e533953a"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.564940 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 00:52:36.568497 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.569945 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:36.570127 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "cf5412dacb3a40cf9d5d16aa51fefba0"
format_stamp: "Formatted at 2026-06-19 00:52:36 on dist-test-slave-r7bh"
server_key: "fa73d364e51bf3745e93d2b39929704a"
server_key_iv: "1067e37284321d181ad5a422e533953a"
server_key_version: "encryptionkey@0"
I20260619 00:52:36.570367 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:36.588295 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:36.592711 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:36.593046 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:36.594062 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:36.595463 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:36.595566 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.595721 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:36.595793 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:36.631919 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45439
I20260619 00:52:36.632007 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45439 every 8 connection(s)
I20260619 00:52:36.633291 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:52:36.639112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 635
I20260619 00:52:36.639317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:52:36.639724 388 external_mini_cluster.cc:1428] Setting key d059f94ecf31d95e74b9f899b3035a60
Jun 19 00:52:36 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:36.655128 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:36.636202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38151 (local address 127.0.97.62:40213)
0619 00:52:36.636402 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:36.636409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:36.638101 (+ 1692us) server_negotiation.cc:407] Connection header received
0619 00:52:36.639987 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:36.639993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:36.640081 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:36.640196 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:36.641936 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.642998 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.644566 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.645358 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.645498 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:36.649280 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:36.649285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:36.649289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:36.649320 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:36.651792 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:36.652775 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:36.652778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:36.652781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:36.652955 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:36.653459 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:36.653463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:36.653465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:36.653705 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:36.653882 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:36.654548 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 00:52:36.654777 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:52:36.656189 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:36.636776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40213 (local address 127.0.97.2:38151)
0619 00:52:36.637924 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:36.637969 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:36.639660 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:36.640413 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:36.640421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:36.641107 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:36.641776 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:36.641797 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.643179 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.643183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:36.644424 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:36.644436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:36.645569 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:36.645575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:36.645767 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:36.646505 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:36.646544 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:36.649052 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:36.651976 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:36.651987 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:36.652008 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:36.652647 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:36.653068 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:36.653076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:36.653082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:36.653346 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:36.653865 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:36.653874 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:36.654199 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 00:52:36.654460 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 00:52:36.654993 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":675,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:52:36.657755 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40213
I20260619 00:52:36.658074 763 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:36.658875 763 heartbeater.cc:511] Master 127.0.97.62:40213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:52:36.660728 440 ts_manager.cc:194] Registered new tserver with Master: cf5412dacb3a40cf9d5d16aa51fefba0 (127.0.97.2:45439)
I20260619 00:52:36.661702 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:52:36.681145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:37.023099 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:37.023417 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:37.023619 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:37.031229 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:37.031365 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:37.031574 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:52:37.041671 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40213
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:37.043071 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:37.044961 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:37.059460 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:37.059420 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:37.059444 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:37.060271 768 server_base.cc:1034] running on GCE node
I20260619 00:52:37.061026 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:37.062119 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:37.063380 768 hybrid_clock.cc:648] HybridClock initialized: now 1781830357063307 us; error 55 us; skew 500 ppm
I20260619 00:52:37.063710 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:37.067343 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:52:37.069062 768 webserver.cc:460] Webserver started at http://127.0.97.3:39641/ using document root <none> and password file <none>
I20260619 00:52:37.069792 768 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:37.069898 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:37.070160 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:37.073325 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "f4617114263c40edbe1376ea25aba273"
format_stamp: "Formatted at 2026-06-19 00:52:37 on dist-test-slave-r7bh"
server_key: "8957c517684f72efe8668337f8142906"
server_key_iv: "f2ca20565c205016b55e7c0c3352195a"
server_key_version: "encryptionkey@0"
I20260619 00:52:37.074249 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f4617114263c40edbe1376ea25aba273"
format_stamp: "Formatted at 2026-06-19 00:52:37 on dist-test-slave-r7bh"
server_key: "8957c517684f72efe8668337f8142906"
server_key_iv: "f2ca20565c205016b55e7c0c3352195a"
server_key_version: "encryptionkey@0"
I20260619 00:52:37.080166 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:52:37.083746 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:37.085179 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:52:37.085348 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "f4617114263c40edbe1376ea25aba273"
format_stamp: "Formatted at 2026-06-19 00:52:37 on dist-test-slave-r7bh"
server_key: "8957c517684f72efe8668337f8142906"
server_key_iv: "f2ca20565c205016b55e7c0c3352195a"
server_key_version: "encryptionkey@0"
I20260619 00:52:37.085543 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:37.122474 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:37.127177 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:37.127424 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:37.128306 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:37.129664 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:37.129739 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:37.129827 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:37.129871 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:37.156327 629 heartbeater.cc:503] Master 127.0.97.62:40213 was elected leader, sending a full tablet report...
I20260619 00:52:37.166076 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40203
I20260619 00:52:37.166117 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40203 every 8 connection(s)
I20260619 00:52:37.167408 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:52:37.168190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 768
I20260619 00:52:37.168402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:52:37.168953 388 external_mini_cluster.cc:1428] Setting key a37def3d426558c5c24ca91dd23e032c
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:37.191879 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.170164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38135 (local address 127.0.97.62:40213)
0619 00:52:37.177235 (+ 7071us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.177240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.177260 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:52:37.177313 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.177318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.177391 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.177493 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.179139 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.179953 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.181651 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.182034 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.182129 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.185761 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.185767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.185776 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.185815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.188766 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.189569 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.189575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.189578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.189702 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.190134 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.190139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.190142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.190442 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.190589 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.191303 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.191508 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6948,"thread_start_us":6840,"threads_started":1}
I20260619 00:52:37.192875 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.170870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40213 (local address 127.0.97.3:38135)
0619 00:52:37.172972 (+ 2102us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:37.173040 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:37.176247 (+ 3207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:37.177742 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:37.177751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:37.178352 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:37.178954 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:37.178972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.180095 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.180099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:37.181511 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:37.181522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.182219 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.182223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:37.182415 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.183057 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:37.183083 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:37.185567 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:37.188972 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:37.188982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:37.189000 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:37.189443 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:37.189813 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:37.189817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:37.189821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:37.190015 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:37.190574 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:37.190582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:37.190925 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 00:52:37.191192 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 00:52:37.191711 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1708,"thread_start_us":530,"threads_started":1}
I20260619 00:52:37.194175 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40213
I20260619 00:52:37.194446 896 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:37.195104 896 heartbeater.cc:511] Master 127.0.97.62:40213 requested a full tablet report, sending...
I20260619 00:52:37.196497 440 ts_manager.cc:194] Registered new tserver with Master: f4617114263c40edbe1376ea25aba273 (127.0.97.3:40203)
I20260619 00:52:37.197387 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38135
I20260619 00:52:37.197818 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:37.244774 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.228326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.0.97.1:36231)
0619 00:52:37.229034 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.229043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.229077 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:52:37.229159 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.229166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.229382 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.229565 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.231012 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.232421 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.233912 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.234492 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.234636 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.238606 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.238617 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.238633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.238672 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.241844 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.242608 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.242616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.242618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.242745 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.243159 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.243164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.243166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.243536 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.243897 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.244151 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.244426 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":310,"threads_started":1}
W20260619 00:52:37.246134 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:37.292922 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.278486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.0.97.1:36231)
0619 00:52:37.278722 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.278728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.278971 (+ 243us) server_negotiation.cc:407] Connection header received
0619 00:52:37.279120 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.279124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.279189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.279308 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.280644 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.281459 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.282790 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.283289 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.283415 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.287462 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.287467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.287471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.287507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.290167 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.290932 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.290940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.290944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.291078 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.291457 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.291463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.291468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.291726 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.291971 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.292298 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.292524 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:52:37.294492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 501
I20260619 00:52:37.307400 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.295442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.0.97.1:36231)
0619 00:52:37.295600 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.295604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.296000 (+ 396us) server_negotiation.cc:407] Connection header received
0619 00:52:37.296172 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.296176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.296232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.296309 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.297557 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.298271 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.299535 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.299974 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.300075 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.302241 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.302246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.302249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.302279 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.304892 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.305474 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.305478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.305481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.306017 (+ 536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.306328 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.306331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.306333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.306572 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.306737 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.306989 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.307149 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 00:52:37.308168 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:37.390566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 501
I20260619 00:52:37.407128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 635
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:52:37.424010 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.408124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.0.97.2:45439)
0619 00:52:37.408635 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.408642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.408668 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:52:37.408735 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.408741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.409040 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.409223 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.410421 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.411434 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.412685 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.413322 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.413472 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.417974 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.417989 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.418005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.418043 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.421069 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.421797 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.421812 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.421814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.421943 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.422371 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.422382 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.422385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.422742 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.423046 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.423367 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.423643 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260619 00:52:37.425452 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:37.506892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 635
I20260619 00:52:37.529251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 768
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:52:37.546500 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:37.530392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.0.97.3:40203)
0619 00:52:37.531046 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:37.531053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:37.531083 (+ 30us) server_negotiation.cc:407] Connection header received
0619 00:52:37.531146 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:37.531152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:37.531364 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:37.531507 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:37.532672 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.533801 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.535315 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:37.535947 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:37.536062 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:37.540554 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:37.540567 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:37.540583 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:37.540622 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:37.543571 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.544265 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.544272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.544275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.544417 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:37.544905 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:37.544909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:37.544911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:37.545314 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:37.545553 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:37.545902 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 00:52:37.546105 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":330,"threads_started":1}
I20260619 00:52:37.547837 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:37.629570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 768
I20260619 00:52:37.645825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 408
I20260619 00:52:37.647003 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:37.749035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 408
2026-06-19T00:52:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6295 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[918](info): set up 2 sockets
Jun 19 00:52:37 dist-test-slave-r7bh krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260619 00:52:38.836781 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:52:38 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:39.021340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38857
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:39.346668 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:39.346982 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:39.347071 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:39.354768 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:52:39.354897 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:39.354947 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:39.355008 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:52:39.355051 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:52:39.365240 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:39.366539 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:39.367827 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:39.379544 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:39.379647 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:39.379776 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:39.380664 931 server_base.cc:1034] running on GCE node
I20260619 00:52:39.381489 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:39.382901 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:39.384141 931 hybrid_clock.cc:648] HybridClock initialized: now 1781830359384102 us; error 53 us; skew 500 ppm
I20260619 00:52:39.384512 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:39 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:39.391258 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:52:39.392764 931 webserver.cc:460] Webserver started at http://127.0.97.62:36623/ using document root <none> and password file <none>
I20260619 00:52:39.393522 931 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:39.393626 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:39.394011 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:39.397123 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "13046b2edcde429389d1b3f2d8bbb739"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "3a669a51d49bbba5ea1d59312391e912"
server_key_iv: "5482a0421ce02e5779efa65f06c8f9a4"
server_key_version: "encryptionkey@0"
I20260619 00:52:39.397913 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "13046b2edcde429389d1b3f2d8bbb739"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "3a669a51d49bbba5ea1d59312391e912"
server_key_iv: "5482a0421ce02e5779efa65f06c8f9a4"
server_key_version: "encryptionkey@0"
I20260619 00:52:39.406261 931 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260619 00:52:39.411367 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:39.413434 931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 00:52:39.413774 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "13046b2edcde429389d1b3f2d8bbb739"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "3a669a51d49bbba5ea1d59312391e912"
server_key_iv: "5482a0421ce02e5779efa65f06c8f9a4"
server_key_version: "encryptionkey@0"
I20260619 00:52:39.414037 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:39.434976 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:39.441236 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:39.441588 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:39.463714 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38857
I20260619 00:52:39.463758 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38857 every 8 connection(s)
I20260619 00:52:39.465276 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:52:39.468852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 931
I20260619 00:52:39.469101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:52:39.469524 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:39.469936 388 external_mini_cluster.cc:1428] Setting key 104cb07bfeb1918fc037731b09bbc338
I20260619 00:52:39.483693 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739: Bootstrap starting.
Jun 19 00:52:39 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:39.487310 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:39.488421 998 log.cc:824] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:39.491528 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739: No bootstrap required, opened a new log
I20260619 00:52:39.492481 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:39.471997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.97.62:38857)
0619 00:52:39.472947 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:39.472962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:39.473010 (+ 48us) server_negotiation.cc:407] Connection header received
0619 00:52:39.474522 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:39.474543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:39.474948 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:39.475279 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:39.477056 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:39.478321 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:39.480077 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:39.480772 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:39.481069 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:39.485100 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:39.485111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:39.485127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:39.485172 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:39.488185 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:39.488997 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:39.489003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:39.489010 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:39.489189 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:39.489637 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:39.489644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:39.489646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:39.490231 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:39.490514 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:39.491011 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 00:52:39.491427 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":383,"threads_started":1}
I20260619 00:52:39.500016 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } }
I20260619 00:52:39.500430 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:39.500536 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13046b2edcde429389d1b3f2d8bbb739, State: Initialized, Role: FOLLOWER
I20260619 00:52:39.501199 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } }
I20260619 00:52:39.501442 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:52:39.501588 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:52:39.501766 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:39.504230 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } }
I20260619 00:52:39.504725 998 leader_election.cc:302] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13046b2edcde429389d1b3f2d8bbb739; no voters:
I20260619 00:52:39.505435 998 leader_election.cc:288] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:52:39.505592 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:39.506278 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [term 1 LEADER]: Becoming Leader. State: Replica: 13046b2edcde429389d1b3f2d8bbb739, State: Running, Role: LEADER
I20260619 00:52:39.506793 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } }
I20260619 00:52:39.507537 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:52:39.509676 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13046b2edcde429389d1b3f2d8bbb739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } } }
I20260619 00:52:39.509811 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13046b2edcde429389d1b3f2d8bbb739. Latest consensus state: current_term: 1 leader_uuid: "13046b2edcde429389d1b3f2d8bbb739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13046b2edcde429389d1b3f2d8bbb739" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38857 } } }
I20260619 00:52:39.510136 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:39.510118 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:39.517475 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:52:39.523339 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:52:39.533069 1011 catalog_manager.cc:1347] Generated new cluster ID: 5243711bab1742a99029b4cb27844fd4
I20260619 00:52:39.533206 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:52:39.546455 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:52:39.548522 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:52:39.556998 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13046b2edcde429389d1b3f2d8bbb739: Generated new TSK 0
I20260619 00:52:39.557930 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:52:39.605473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:39.939601 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:39.939988 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:39.940187 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:39.947814 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:39.947970 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:39.948170 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:52:39.958563 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:39.960085 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:39.961588 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:39.975558 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:39.975479 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:39.975521 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:39.976356 1024 server_base.cc:1034] running on GCE node
I20260619 00:52:39.977043 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:39.977962 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:39.979161 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781830359979116 us; error 37 us; skew 500 ppm
I20260619 00:52:39.979490 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:39 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:39.982700 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:52:39.984233 1024 webserver.cc:460] Webserver started at http://127.0.97.1:34477/ using document root <none> and password file <none>
I20260619 00:52:39.985005 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:39.985139 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:39.985469 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:39.988382 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "6abe19447ea5449398f09062a4afecb5"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "004137c06963c8ad489a2208ddfc2912"
server_key_iv: "a639aa7f5b4c36fda009e9c85b272a93"
server_key_version: "encryptionkey@0"
I20260619 00:52:39.989226 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6abe19447ea5449398f09062a4afecb5"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "004137c06963c8ad489a2208ddfc2912"
server_key_iv: "a639aa7f5b4c36fda009e9c85b272a93"
server_key_version: "encryptionkey@0"
I20260619 00:52:39.995009 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:52:39.998591 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:39.999922 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:52:40.000102 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "6abe19447ea5449398f09062a4afecb5"
format_stamp: "Formatted at 2026-06-19 00:52:39 on dist-test-slave-r7bh"
server_key: "004137c06963c8ad489a2208ddfc2912"
server_key_iv: "a639aa7f5b4c36fda009e9c85b272a93"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.000314 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:40.037364 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:40.041728 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:40.042030 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:40.043017 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:40.044363 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:40.044462 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.044628 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:40.044701 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.077838 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45963
I20260619 00:52:40.077885 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45963 every 8 connection(s)
I20260619 00:52:40.079128 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:52:40.085242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1024
I20260619 00:52:40.085477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:52:40.086079 388 external_mini_cluster.cc:1428] Setting key 2a6b1dea4349e28762b00822f7d60338
Jun 19 00:52:40 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:40.100606 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:40.081824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58533 (local address 127.0.97.62:38857)
0619 00:52:40.082273 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:40.082277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:40.083174 (+ 897us) server_negotiation.cc:407] Connection header received
0619 00:52:40.085056 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:40.085061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:40.085151 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:40.085271 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:40.086880 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.087856 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.089485 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.089886 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.090018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:40.093636 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:40.093645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:40.093654 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:40.093699 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:40.097282 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:40.098019 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:40.098027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:40.098033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:40.098173 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:40.098574 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:40.098582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:40.098588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:40.098997 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:40.099155 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:40.099878 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 00:52:40.100140 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260619 00:52:40.101444 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:40.082302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38857 (local address 127.0.97.1:58533)
0619 00:52:40.083008 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:40.083051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:40.084725 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:40.085525 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:40.085534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:40.086080 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:40.086700 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:40.086717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.088022 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.088026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:40.089330 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:40.089341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.090126 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.090133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:40.090359 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:40.091053 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:40.091079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:40.093373 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:40.097422 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:40.097429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:40.097442 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:40.097859 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:40.098274 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:40.098277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:40.098279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:40.098450 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:40.099131 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:40.099137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:40.099468 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 00:52:40.099689 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 00:52:40.100077 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":284,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:52:40.103586 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38857
I20260619 00:52:40.103924 1152 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:40.104849 1152 heartbeater.cc:511] Master 127.0.97.62:38857 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:52:40.107062 963 ts_manager.cc:194] Registered new tserver with Master: 6abe19447ea5449398f09062a4afecb5 (127.0.97.1:45963)
I20260619 00:52:40.109373 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:52:40.128242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:40.445212 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:40.445501 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:40.445690 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:40.452962 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:40.453073 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:40.453277 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:52:40.463737 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:40.465090 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:40.466432 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:40.479913 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:40.479977 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:40.479948 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:40.480703 1158 server_base.cc:1034] running on GCE node
I20260619 00:52:40.481473 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:40.482497 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:40.483706 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781830360483638 us; error 56 us; skew 500 ppm
I20260619 00:52:40.484046 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:40 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:40.490164 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:52:40.491734 1158 webserver.cc:460] Webserver started at http://127.0.97.2:42601/ using document root <none> and password file <none>
I20260619 00:52:40.492465 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:40.492568 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:40.492908 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:40.495924 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "eae23ceb57ba4054b084938fb81015e6"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "1a152fbe0e99e810c72b014cd207d929"
server_key_iv: "f4549ca1c83e90db0b484bf8ce314469"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.496657 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eae23ceb57ba4054b084938fb81015e6"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "1a152fbe0e99e810c72b014cd207d929"
server_key_iv: "f4549ca1c83e90db0b484bf8ce314469"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.501984 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 00:52:40.505712 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.507052 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:52:40.507216 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "eae23ceb57ba4054b084938fb81015e6"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "1a152fbe0e99e810c72b014cd207d929"
server_key_iv: "f4549ca1c83e90db0b484bf8ce314469"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.507457 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:40.526083 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:40.530290 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:40.530596 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:40.531658 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:40.533211 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:40.533344 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.533499 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:40.533571 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.565863 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42699
I20260619 00:52:40.565904 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42699 every 8 connection(s)
I20260619 00:52:40.567142 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:52:40.574855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1158
I20260619 00:52:40.575073 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:52:40.575485 388 external_mini_cluster.cc:1428] Setting key 303f059424b3c23aed012b66f82df303
Jun 19 00:52:40 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:40.588392 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:40.569530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39399 (local address 127.0.97.62:38857)
0619 00:52:40.569711 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:40.569715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:40.571010 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 00:52:40.572864 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:40.572872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:40.572957 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:40.573054 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:40.574513 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.575344 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.577506 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.577934 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.578066 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:40.581815 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:40.581822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:40.581828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:40.581882 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:40.584936 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:40.585992 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:40.585996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:40.585999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:40.586125 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:40.586640 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:40.586643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:40.586645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:40.586919 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:40.587028 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:40.587989 (+ 961us) server_negotiation.cc:299] Negotiation successful
0619 00:52:40.588156 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:52:40.590139 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:40.570028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38857 (local address 127.0.97.2:39399)
0619 00:52:40.570830 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:40.570891 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:40.572532 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:40.573274 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:40.573283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:40.573777 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:40.574343 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:40.574359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.575506 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.575511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:40.577359 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:40.577372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:40.578174 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:40.578182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:40.578411 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:40.579137 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:40.579165 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:40.581567 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:40.585105 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:40.585118 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:40.585141 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:40.585848 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:40.586245 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:40.586253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:40.586260 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:40.586527 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:40.587052 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:40.587063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:40.587577 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 00:52:40.587907 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 00:52:40.588542 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":336,"threads_started":1}
I20260619 00:52:40.591732 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38857
I20260619 00:52:40.592062 1286 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:40.592774 1286 heartbeater.cc:511] Master 127.0.97.62:38857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:52:40.594327 963 ts_manager.cc:194] Registered new tserver with Master: eae23ceb57ba4054b084938fb81015e6 (127.0.97.2:42699)
I20260619 00:52:40.595081 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:52:40.616631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:40.933861 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:40.934201 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:40.934378 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:40.941795 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:40.941973 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:40.942153 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:52:40.952219 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38857
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:40.953577 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:40.954948 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:40.968189 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:40.968293 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:40.968328 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:40.969239 1291 server_base.cc:1034] running on GCE node
I20260619 00:52:40.969869 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:40.970872 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:40.972103 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781830360972040 us; error 61 us; skew 500 ppm
I20260619 00:52:40.972415 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:40 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:40.975613 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:52:40.977231 1291 webserver.cc:460] Webserver started at http://127.0.97.3:36115/ using document root <none> and password file <none>
I20260619 00:52:40.977919 1291 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:40.978061 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:40.978371 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:40.981293 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "6d4e7e9393a94794bae10970e7698dc7"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "a7cc9950017db10d8a53bf48382faf71"
server_key_iv: "17a5bffd117b03210bcdba933288282b"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.982079 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4e7e9393a94794bae10970e7698dc7"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "a7cc9950017db10d8a53bf48382faf71"
server_key_iv: "17a5bffd117b03210bcdba933288282b"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.987241 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:52:40.990782 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:40.992213 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:52:40.992383 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "6d4e7e9393a94794bae10970e7698dc7"
format_stamp: "Formatted at 2026-06-19 00:52:40 on dist-test-slave-r7bh"
server_key: "a7cc9950017db10d8a53bf48382faf71"
server_key_iv: "17a5bffd117b03210bcdba933288282b"
server_key_version: "encryptionkey@0"
I20260619 00:52:40.992584 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:41.047585 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:41.052047 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:41.052357 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:41.053423 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:41.054761 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:41.054843 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:41.055014 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:41.055068 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:41.087109 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46799
I20260619 00:52:41.087179 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46799 every 8 connection(s)
I20260619 00:52:41.088356 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:52:41.093675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1291
I20260619 00:52:41.093904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:52:41.094327 388 external_mini_cluster.cc:1428] Setting key 8de6b37a2b579b27a07995621205855b
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:41.110751 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.091079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38687 (local address 127.0.97.62:38857)
0619 00:52:41.091522 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.091527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.092483 (+ 956us) server_negotiation.cc:407] Connection header received
0619 00:52:41.094433 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.094438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.094532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.094652 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.096354 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.097606 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.099143 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.099555 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.099654 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.103202 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.103208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.103213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.103257 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.107180 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.108238 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.108244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.108247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.108402 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.108876 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.108881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.108885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.109234 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.109383 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.110180 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.110379 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260619 00:52:41.111807 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.091642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38857 (local address 127.0.97.3:38687)
0619 00:52:41.092306 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.092354 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.094158 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.094895 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.094903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.095415 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:41.096132 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.096157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.097730 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.097734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.099009 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.099020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.099775 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.099783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.100035 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.100707 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:41.100730 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:41.102974 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:41.107358 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:41.107368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:41.107386 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:41.108003 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:41.108505 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:41.108510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:41.108514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:41.108710 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:41.109361 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:41.109369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:41.109754 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.110101 (+ 347us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.110639 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":261,"threads_started":1}
I20260619 00:52:41.112353 1152 heartbeater.cc:503] Master 127.0.97.62:38857 was elected leader, sending a full tablet report...
I20260619 00:52:41.113201 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38857
I20260619 00:52:41.113523 1419 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:41.114295 1419 heartbeater.cc:511] Master 127.0.97.62:38857 requested a full tablet report, sending...
I20260619 00:52:41.115707 963 ts_manager.cc:194] Registered new tserver with Master: 6d4e7e9393a94794bae10970e7698dc7 (127.0.97.3:46799)
I20260619 00:52:41.116462 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38687
I20260619 00:52:41.122700 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:52:41.142643 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.130200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.0.97.62:38857)
0619 00:52:41.130360 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.130365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.130829 (+ 464us) server_negotiation.cc:407] Connection header received
0619 00:52:41.131022 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.131026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.131110 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.131205 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.132593 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.133415 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.134642 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.135087 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.135213 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.137645 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.137650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.137653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.137683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.140078 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.140694 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.140702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.140709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.140867 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.141228 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.141235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.141238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.141548 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.141789 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.142112 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.142317 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 00:52:41.149485 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:41.156278 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:41.193853 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.169369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45963 (local address 127.0.0.1:36832)
0619 00:52:41.184712 (+ 15343us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.184731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.184932 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.185785 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.185794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.185824 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:41.186246 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.186259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.188407 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.188412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.191080 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.191098 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.192696 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.192701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.192911 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.192970 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.193143 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.193360 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15224,"mutex_wait_us":25,"spinlock_wait_cycles":13952,"thread_start_us":15066,"threads_started":1}
I20260619 00:52:41.194854 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.169698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.97.1:45963)
0619 00:52:41.184340 (+ 14642us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.184351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.185129 (+ 778us) server_negotiation.cc:407] Connection header received
0619 00:52:41.185232 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.185243 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.185554 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.186439 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:41.186652 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.188254 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.191289 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.192497 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.192654 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.192914 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.194379 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.194524 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14446,"thread_start_us":14251,"threads_started":1}
I20260619 00:52:41.195428 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.168912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46799 (local address 127.0.0.1:49066)
0619 00:52:41.184604 (+ 15692us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.184671 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.184935 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.186552 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.186558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.186579 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:41.186994 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.187009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.188892 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.188897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.191753 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.191767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.194557 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.194565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.194744 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.194777 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.194864 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.194992 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15338,"mutex_wait_us":728,"spinlock_wait_cycles":11904,"thread_start_us":16194,"threads_started":1}
I20260619 00:52:41.195448 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.169115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.0.97.3:46799)
0619 00:52:41.185723 (+ 16608us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.185734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.185765 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:52:41.185849 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.185858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.186092 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.186282 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:41.187213 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.188690 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.191911 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.194077 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.194203 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.194429 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.194948 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.195123 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16372,"thread_start_us":16224,"threads_started":1}
I20260619 00:52:41.199060 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2 (DEFAULT_TABLE table=test-table [id=5c3a38737ccb4643a4bdbfccc0a36a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:41.199060 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2 (DEFAULT_TABLE table=test-table [id=5c3a38737ccb4643a4bdbfccc0a36a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:41.199662 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.185705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.0.1:50470)
0619 00:52:41.191382 (+ 5677us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.191401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.191498 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.193039 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.193044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.193068 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:41.193439 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.193450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.195327 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.195333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.197539 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.197553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.198869 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.198875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.199032 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.199065 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.199185 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.199317 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5555,"thread_start_us":5419,"threads_started":1}
I20260619 00:52:41.199693 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.185748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.0.97.2:42699)
0619 00:52:41.191722 (+ 5974us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.191731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.191763 (+ 32us) server_negotiation.cc:407] Connection header received
0619 00:52:41.191873 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.191880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.192109 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.192767 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:41.193648 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.195168 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.197763 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.198672 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.198823 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.199067 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.199236 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.199391 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5842,"thread_start_us":5736,"threads_started":1}
I20260619 00:52:41.200662 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:41.200960 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:41.202868 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2 (DEFAULT_TABLE table=test-table [id=5c3a38737ccb4643a4bdbfccc0a36a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:41.204197 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:41.215935 1438 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Bootstrap starting.
I20260619 00:52:41.218000 1439 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Bootstrap starting.
I20260619 00:52:41.219707 1440 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Bootstrap starting.
I20260619 00:52:41.220654 1438 tablet_bootstrap.cc:654] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:41.222684 1438 log.cc:824] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:41.223305 1439 tablet_bootstrap.cc:654] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:41.223774 1440 tablet_bootstrap.cc:654] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:41.224310 1439 log.cc:824] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:41.225229 1440 log.cc:824] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:41.229803 1440 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: No bootstrap required, opened a new log
I20260619 00:52:41.230100 1440 ts_tablet_manager.cc:1397] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 00:52:41.230469 1439 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: No bootstrap required, opened a new log
I20260619 00:52:41.230715 1439 ts_tablet_manager.cc:1397] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260619 00:52:41.232345 1438 tablet_bootstrap.cc:492] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: No bootstrap required, opened a new log
I20260619 00:52:41.232610 1438 ts_tablet_manager.cc:1397] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260619 00:52:41.237926 1439 raft_consensus.cc:348] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.238303 1439 raft_consensus.cc:374] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:41.238390 1439 raft_consensus.cc:729] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abe19447ea5449398f09062a4afecb5, State: Initialized, Role: FOLLOWER
I20260619 00:52:41.238987 1439 consensus_queue.cc:260] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.240720 1439 ts_tablet_manager.cc:1428] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 00:52:41.241027 1440 raft_consensus.cc:348] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.241438 1440 raft_consensus.cc:374] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:41.241535 1440 raft_consensus.cc:729] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4e7e9393a94794bae10970e7698dc7, State: Initialized, Role: FOLLOWER
I20260619 00:52:41.242141 1440 consensus_queue.cc:260] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.243260 1419 heartbeater.cc:503] Master 127.0.97.62:38857 was elected leader, sending a full tablet report...
I20260619 00:52:41.243875 1440 ts_tablet_manager.cc:1428] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 00:52:41.244588 1438 raft_consensus.cc:348] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.245048 1438 raft_consensus.cc:374] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:41.245154 1438 raft_consensus.cc:729] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae23ceb57ba4054b084938fb81015e6, State: Initialized, Role: FOLLOWER
I20260619 00:52:41.245719 1438 consensus_queue.cc:260] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.247021 1286 heartbeater.cc:503] Master 127.0.97.62:38857 was elected leader, sending a full tablet report...
I20260619 00:52:41.247462 1438 ts_tablet_manager.cc:1428] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 00:52:41.319139 1287 tablet.cc:2369] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:41.325603 1445 raft_consensus.cc:482] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:41.325856 1445 raft_consensus.cc:504] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.327462 1445 leader_election.cc:288] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eae23ceb57ba4054b084938fb81015e6 (127.0.97.2:42699), 6abe19447ea5449398f09062a4afecb5 (127.0.97.1:45963)
W20260619 00:52:41.331559 1153 tablet.cc:2369] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:41.334236 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.327780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57145 (local address 127.0.97.2:42699)
0619 00:52:41.327980 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.327987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.328008 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:52:41.328201 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.328210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.328311 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.328417 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:41.328881 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.329845 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.332051 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.333318 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.333467 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.333567 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.333891 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.334024 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:52:41.334300 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.327640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.97.3:57145)
0619 00:52:41.327882 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.327902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.328033 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.328486 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.328490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.328516 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:41.328729 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.328739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.329973 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.329977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.331893 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.331906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.333499 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.333504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.333646 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.333671 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.333857 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.334004 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 00:52:41.334602 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.328165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37971 (local address 127.0.97.1:45963)
0619 00:52:41.328381 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.328387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.328742 (+ 355us) server_negotiation.cc:407] Connection header received
0619 00:52:41.328932 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.328938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.329014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.329103 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:41.329538 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.330372 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.332479 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.333670 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.333813 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.333927 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.334201 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.334351 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 00:52:41.334832 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.327852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45963 (local address 127.0.97.3:37971)
0619 00:52:41.328622 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.328644 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:41.328760 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:41.329124 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:41.329127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:41.329145 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:41.329398 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.329404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.330517 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.330522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.332329 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:41.332338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.333884 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.333890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:41.334047 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.334076 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:52:41.334166 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 00:52:41.334298 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":432,"threads_started":1}
I20260619 00:52:41.335364 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbb3d68f8ab41efb4759ace8fdb6df2" candidate_uuid: "6d4e7e9393a94794bae10970e7698dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae23ceb57ba4054b084938fb81015e6" is_pre_election: true
I20260619 00:52:41.335951 1241 raft_consensus.cc:2454] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4e7e9393a94794bae10970e7698dc7 in term 0.
I20260619 00:52:41.335956 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbb3d68f8ab41efb4759ace8fdb6df2" candidate_uuid: "6d4e7e9393a94794bae10970e7698dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abe19447ea5449398f09062a4afecb5" is_pre_election: true
I20260619 00:52:41.336531 1107 raft_consensus.cc:2454] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4e7e9393a94794bae10970e7698dc7 in term 0.
I20260619 00:52:41.336750 1310 leader_election.cc:302] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4e7e9393a94794bae10970e7698dc7, eae23ceb57ba4054b084938fb81015e6; no voters:
I20260619 00:52:41.337229 1445 raft_consensus.cc:2781] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:41.337379 1445 raft_consensus.cc:482] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:41.337487 1445 raft_consensus.cc:3037] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 0 FOLLOWER]: Advancing to term 1
W20260619 00:52:41.339972 1420 tablet.cc:2369] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:41.339990 1445 raft_consensus.cc:504] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.340718 1445 leader_election.cc:288] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [CANDIDATE]: Term 1 election: Requested vote from peers eae23ceb57ba4054b084938fb81015e6 (127.0.97.2:42699), 6abe19447ea5449398f09062a4afecb5 (127.0.97.1:45963)
I20260619 00:52:41.341161 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbb3d68f8ab41efb4759ace8fdb6df2" candidate_uuid: "6d4e7e9393a94794bae10970e7698dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae23ceb57ba4054b084938fb81015e6"
I20260619 00:52:41.341199 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbb3d68f8ab41efb4759ace8fdb6df2" candidate_uuid: "6d4e7e9393a94794bae10970e7698dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abe19447ea5449398f09062a4afecb5"
I20260619 00:52:41.341362 1241 raft_consensus.cc:3037] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:41.341383 1107 raft_consensus.cc:3037] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:41.343468 1241 raft_consensus.cc:2454] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P eae23ceb57ba4054b084938fb81015e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4e7e9393a94794bae10970e7698dc7 in term 1.
I20260619 00:52:41.343472 1107 raft_consensus.cc:2454] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6abe19447ea5449398f09062a4afecb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4e7e9393a94794bae10970e7698dc7 in term 1.
I20260619 00:52:41.343869 1310 leader_election.cc:302] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4e7e9393a94794bae10970e7698dc7, eae23ceb57ba4054b084938fb81015e6; no voters:
I20260619 00:52:41.344189 1445 raft_consensus.cc:2781] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:41.344583 1445 raft_consensus.cc:686] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4e7e9393a94794bae10970e7698dc7, State: Running, Role: LEADER
I20260619 00:52:41.345114 1445 consensus_queue.cc:237] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } }
I20260619 00:52:41.349681 963 catalog_manager.cc:5495] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4e7e9393a94794bae10970e7698dc7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6d4e7e9393a94794bae10970e7698dc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4e7e9393a94794bae10970e7698dc7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae23ceb57ba4054b084938fb81015e6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6abe19447ea5449398f09062a4afecb5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45963 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:41.378448 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.363203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.97.1:45963)
0619 00:52:41.363403 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.363408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.363958 (+ 550us) server_negotiation.cc:407] Connection header received
0619 00:52:41.364184 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.364189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.364265 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.364396 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.366012 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.366777 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.368087 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.368523 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.368676 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.372970 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.372986 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.373006 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.373042 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.375659 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.376365 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.376381 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.376383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.376519 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.376919 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.376926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.376931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.377281 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.377589 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.377871 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.378087 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:41.424173 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.410159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.0.97.1:45963)
0619 00:52:41.410373 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.410379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.410751 (+ 372us) server_negotiation.cc:407] Connection header received
0619 00:52:41.410958 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.410962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.411025 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.411135 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.412475 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.413291 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.414543 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.414964 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.415097 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.419037 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.419043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.419046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.419077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.421776 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.422407 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.422413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.422417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.422639 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.423038 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.423041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.423043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.423307 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.423516 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.423740 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.423933 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:41.469635 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.455446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.0.97.1:45963)
0619 00:52:41.455702 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.455707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.455805 (+ 98us) server_negotiation.cc:407] Connection header received
0619 00:52:41.455973 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.455977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.456039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.456127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.457990 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.458706 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.459921 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.460341 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.460456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.464401 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.464406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.464409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.464442 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.466998 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.467766 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.467770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.467772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.467916 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.468309 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.468313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.468315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.468610 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.468921 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.469179 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.469359 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 00:52:41.471426 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4bbb3d68f8ab41efb4759ace8fdb6df2: Uninitialized: clean time has not yet been initialized
I20260619 00:52:41.474611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1024
I20260619 00:52:41.487577 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.475935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.97.1:45963)
0619 00:52:41.476111 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.476117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.476484 (+ 367us) server_negotiation.cc:407] Connection header received
0619 00:52:41.476617 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.476622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.476703 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.476842 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.477969 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.479015 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.480541 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.481010 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.481115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.483165 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.483169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.483172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.483195 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.485499 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.486041 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.486044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.486047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.486156 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.486463 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.486466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.486468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.486708 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.486930 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.487193 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.487350 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:52:41.488335 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:41.583052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1024
I20260619 00:52:41.599779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1158
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:52:41.614925 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.600565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.0.97.2:42699)
0619 00:52:41.600739 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.600746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.600768 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:52:41.601034 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.601039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.601133 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.601242 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.602393 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.603176 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.604448 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.604915 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.605062 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.609259 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.609274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.609296 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.609330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.612021 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.612706 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.612713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.612715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.612892 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.613270 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.613277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.613280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.613664 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.613982 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.614264 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.614499 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:52:41.615680 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:41.703822 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1158
I20260619 00:52:41.721093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1291
I20260619 00:52:41.728713 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.728238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.97.3:33447)
0619 00:52:41.728471 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.728582 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 00:52:41.729096 1310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.2:42699: connect: Connection refused (error 111)
Jun 19 00:52:41 dist-test-slave-r7bh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:52:41.730517 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.730172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42699 (local address 127.0.97.3:33449)
0619 00:52:41.730313 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:41.730385 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:42699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:52:41.730751 1310 consensus_peers.cc:487] T 4bbb3d68f8ab41efb4759ace8fdb6df2 P 6d4e7e9393a94794bae10970e7698dc7 -> Peer eae23ceb57ba4054b084938fb81015e6 (127.0.97.2:42699): Couldn't send request to peer eae23ceb57ba4054b084938fb81015e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.2:42699: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 00:52:41.735940 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:41.721846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.97.3:46799)
0619 00:52:41.722304 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:41.722309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:41.722325 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:52:41.722380 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:41.722383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:41.722458 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:41.722580 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:41.723875 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.724599 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.725834 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:41.726213 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:41.726304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:41.730852 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:41.730865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:41.730882 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:41.730930 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:41.733524 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.734122 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.734129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.734131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.734248 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:41.734591 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:41.734593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:41.734595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:41.734918 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:41.735113 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:41.735379 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 00:52:41.735544 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":218,"threads_started":1}
I20260619 00:52:41.736588 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:41.827234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1291
I20260619 00:52:41.843804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 931
I20260619 00:52:41.844985 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:41.951473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 931
2026-06-19T00:52:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4203 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:42 dist-test-slave-r7bh krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:42 dist-test-slave-r7bh krb5kdc[1469](info): set up 2 sockets
Jun 19 00:52:42 dist-test-slave-r7bh krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260619 00:52:43.096706 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:52:43 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:43.278081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:43.597251 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:43.597536 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:43.597623 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:43.604699 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:52:43.604802 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 00:52:43.604866 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:43.604929 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:43.604975 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:52:43.605036 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:52:43.614627 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42755
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:43.615815 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:43.617152 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:43.628671 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:43.628753 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:43.628726 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:43.629791 1482 server_base.cc:1034] running on GCE node
I20260619 00:52:43.630554 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:43.631904 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:43.633195 1482 hybrid_clock.cc:648] HybridClock initialized: now 1781830363633117 us; error 40 us; skew 500 ppm
I20260619 00:52:43.633611 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:43 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:43.637003 1482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:52:43.638561 1482 webserver.cc:460] Webserver started at http://127.0.97.62:33123/ using document root <none> and password file <none>
I20260619 00:52:43.639289 1482 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:43.639425 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:43.639758 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:43.642871 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "51162d3d3461419d8a56c19129ac9cd6"
format_stamp: "Formatted at 2026-06-19 00:52:43 on dist-test-slave-r7bh"
server_key: "a06f5e62a0334ee9c21810f023a3c205"
server_key_iv: "ed60a4b34aa9c7a98ec23b95af470dcf"
server_key_version: "encryptionkey@0"
I20260619 00:52:43.643699 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "51162d3d3461419d8a56c19129ac9cd6"
format_stamp: "Formatted at 2026-06-19 00:52:43 on dist-test-slave-r7bh"
server_key: "a06f5e62a0334ee9c21810f023a3c205"
server_key_iv: "ed60a4b34aa9c7a98ec23b95af470dcf"
server_key_version: "encryptionkey@0"
I20260619 00:52:43.648895 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:52:43.652308 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:43.653736 1482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:43.653918 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "51162d3d3461419d8a56c19129ac9cd6"
format_stamp: "Formatted at 2026-06-19 00:52:43 on dist-test-slave-r7bh"
server_key: "a06f5e62a0334ee9c21810f023a3c205"
server_key_iv: "ed60a4b34aa9c7a98ec23b95af470dcf"
server_key_version: "encryptionkey@0"
I20260619 00:52:43.654138 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:43.672071 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:43.676348 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:43.676640 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:43.697863 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42755
I20260619 00:52:43.697870 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42755 every 8 connection(s)
I20260619 00:52:43.699261 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:52:43.702963 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:43.705601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1482
I20260619 00:52:43.705811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:52:43.706593 388 external_mini_cluster.cc:1428] Setting key 8a4574488a1964c3e8323ada0989e82f
I20260619 00:52:43.714670 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Bootstrap starting.
I20260619 00:52:43.718230 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:43.719228 1559 log.cc:824] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:52:43 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:43.722400 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: No bootstrap required, opened a new log
I20260619 00:52:43.730304 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:43.708703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.97.62:42755)
0619 00:52:43.709470 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:43.709483 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:43.709522 (+ 39us) server_negotiation.cc:407] Connection header received
0619 00:52:43.711198 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:43.711221 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:43.711639 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:43.712050 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:43.713541 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:43.714900 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:43.716164 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:43.716996 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:43.717182 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:43.721378 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:43.721393 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:43.721413 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:43.721459 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:43.725475 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:43.726366 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:43.726375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:43.726384 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:43.726593 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:43.726959 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:43.726964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:43.726967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:43.727665 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:43.727973 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:43.728427 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 00:52:43.728905 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":314,"threads_started":1}
I20260619 00:52:43.732944 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260619 00:52:43.733378 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:43.733481 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51162d3d3461419d8a56c19129ac9cd6, State: Initialized, Role: FOLLOWER
I20260619 00:52:43.734084 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260619 00:52:43.734313 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:52:43.734458 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:52:43.734628 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:43.736775 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260619 00:52:43.737335 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51162d3d3461419d8a56c19129ac9cd6; no voters:
I20260619 00:52:43.737969 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:52:43.738080 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:43.739290 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 51162d3d3461419d8a56c19129ac9cd6, State: Running, Role: LEADER
I20260619 00:52:43.739780 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260619 00:52:43.740213 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:52:43.742646 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51162d3d3461419d8a56c19129ac9cd6. Latest consensus state: current_term: 1 leader_uuid: "51162d3d3461419d8a56c19129ac9cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } } }
I20260619 00:52:43.743041 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:43.743009 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51162d3d3461419d8a56c19129ac9cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51162d3d3461419d8a56c19129ac9cd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } } }
I20260619 00:52:43.743295 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:43.747138 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:52:43.752534 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:52:43.761881 1572 catalog_manager.cc:1347] Generated new cluster ID: 6d828b91d1bf45aaad6fd688be205b38
I20260619 00:52:43.762035 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:52:43.780030 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:52:43.781454 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:52:43.788743 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Generated new TSK 0
I20260619 00:52:43.789597 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:52:43.838757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:44.171048 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:44.171320 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:44.171530 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:44.178759 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:44.178861 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:44.179020 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:44.179141 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:52:44.189050 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:44.190291 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:44.191596 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:44.203506 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:44.204247 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:44.205168 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:44.205425 1585 server_base.cc:1034] running on GCE node
I20260619 00:52:44.205978 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:44.206861 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:44.208070 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781830364208014 us; error 43 us; skew 500 ppm
I20260619 00:52:44.208371 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:44 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:44.211616 1585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:52:44.213127 1585 webserver.cc:460] Webserver started at http://127.0.97.1:43763/ using document root <none> and password file <none>
I20260619 00:52:44.213843 1585 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:44.213929 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:44.214264 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:44.217363 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "9edc4d49688c41659b9fce7b52d2b3e2"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "063bc31c685471e159c5d8fe383b9ae5"
server_key_iv: "a9bdd449fbde7d396c4ab632c5935877"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.218158 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9edc4d49688c41659b9fce7b52d2b3e2"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "063bc31c685471e159c5d8fe383b9ae5"
server_key_iv: "a9bdd449fbde7d396c4ab632c5935877"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.223315 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 00:52:44.226807 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:44.228173 1585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:52:44.228358 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "9edc4d49688c41659b9fce7b52d2b3e2"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "063bc31c685471e159c5d8fe383b9ae5"
server_key_iv: "a9bdd449fbde7d396c4ab632c5935877"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.228574 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:44.249630 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:44.253515 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:44.253831 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:44.256600 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:44.256708 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:44.256861 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:44.256966 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 19 00:52:44 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:44.280591 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.257153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39965 (local address 127.0.97.62:42755)
0619 00:52:44.257987 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:44.257991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:44.259012 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 00:52:44.261027 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:44.261033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:44.261123 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:44.261241 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:44.263758 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.265057 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.267590 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.268218 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.268357 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.274086 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:44.274092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:44.274095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:44.274130 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:44.276688 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.277830 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.277835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.277837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.277947 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.278525 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.278529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.278531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.278841 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:44.278951 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:44.280120 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0619 00:52:44.280351 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":619,"threads_started":1}
I20260619 00:52:44.282809 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.257928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.1:39965)
0619 00:52:44.258809 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:44.258846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:44.260702 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:44.261508 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:44.261516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:44.262406 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:44.263469 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.263489 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.265213 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.265221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.267433 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.267445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.268481 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.268488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.268703 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.269531 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:44.269588 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:44.273786 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:44.276869 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.276883 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.276908 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.277686 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.278060 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.278068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.278074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.278382 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.278972 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:44.278984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:44.279627 (+ 643us) client_negotiation.cc:769] Sending connection context
0619 00:52:44.280000 (+ 373us) client_negotiation.cc:240] Negotiation successful
0619 00:52:44.280743 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":380,"threads_started":1}
I20260619 00:52:44.306670 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.289754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38629 (local address 127.0.97.62:42755)
0619 00:52:44.289992 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:44.290001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:44.290581 (+ 580us) server_negotiation.cc:407] Connection header received
0619 00:52:44.290813 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:44.290821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:44.290880 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:44.290957 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:44.292437 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.293356 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.295987 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.296377 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.296469 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.299467 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:44.299474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:44.299479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:44.299522 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:44.303121 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.304255 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.304262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.304268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.304429 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.304920 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.304926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.304930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.305224 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:44.305333 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:44.305980 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 00:52:44.306243 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:52:44.307142 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.289720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.1:38629)
0619 00:52:44.290462 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:44.290487 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:44.290674 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:44.291019 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:44.291026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:44.291834 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:44.292288 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.292303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.293509 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.293516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.295827 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.295844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.296580 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.296590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.296870 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.297707 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:44.297768 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:44.299253 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:44.303283 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.303292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.303299 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.304094 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.304568 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.304573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.304577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.304774 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.305345 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:44.305353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:44.305649 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 00:52:44.305865 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 00:52:44.306343 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":383,"threads_started":1}
I20260619 00:52:44.311200 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33567
I20260619 00:52:44.311249 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33567 every 8 connection(s)
I20260619 00:52:44.312477 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:52:44.314615 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260619 00:52:44.314859 1721 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:44.315524 1721 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260619 00:52:44.317756 1514 ts_manager.cc:194] Registered new tserver with Master: 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1:33567)
I20260619 00:52:44.318485 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1585
I20260619 00:52:44.318727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:52:44.319483 388 external_mini_cluster.cc:1428] Setting key 2c11e936427e5bcb73eff2d41211b0cf
I20260619 00:52:44.319522 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39965
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:52:44.359972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:44.685550 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:44.685822 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:44.685916 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:44.692973 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:44.693075 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:44.693157 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:44.693238 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:52:44.702767 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:44.704010 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:44.705417 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:44.718576 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:44.718539 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:44.718505 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:44.719497 1725 server_base.cc:1034] running on GCE node
I20260619 00:52:44.720139 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:44.721135 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:44.722397 1725 hybrid_clock.cc:648] HybridClock initialized: now 1781830364722343 us; error 33 us; skew 500 ppm
I20260619 00:52:44.722738 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:44 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:44.726167 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:52:44.727614 1725 webserver.cc:460] Webserver started at http://127.0.97.2:38581/ using document root <none> and password file <none>
I20260619 00:52:44.728324 1725 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:44.728439 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:44.728793 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:44.732029 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "7a6befbc93174f3393b1ae3a85b84324"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "8901306c4d50751df7c8dd56a8cb4d86"
server_key_iv: "ce311d9e270827956e71e88c4a582e79"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.732977 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7a6befbc93174f3393b1ae3a85b84324"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "8901306c4d50751df7c8dd56a8cb4d86"
server_key_iv: "ce311d9e270827956e71e88c4a582e79"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.738248 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:52:44.741988 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:44.743455 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:52:44.743646 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "7a6befbc93174f3393b1ae3a85b84324"
format_stamp: "Formatted at 2026-06-19 00:52:44 on dist-test-slave-r7bh"
server_key: "8901306c4d50751df7c8dd56a8cb4d86"
server_key_iv: "ce311d9e270827956e71e88c4a582e79"
server_key_version: "encryptionkey@0"
I20260619 00:52:44.743891 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:44.763818 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:44.769289 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:44.769606 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:44.778286 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:44.778395 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:44.778542 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:44.778622 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:52:44 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:44.804085 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.779570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.2:35227)
0619 00:52:44.780417 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:44.780472 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:44.782593 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:44.783518 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:44.783533 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:44.784307 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:44.785346 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.785366 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.786692 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.786700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.789116 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.789134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.790229 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.790240 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.790456 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.791207 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:44.791230 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:44.794157 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:44.798220 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.798229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.798246 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.798900 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.799360 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.799365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.799369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.799631 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.800376 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:44.800385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:44.800959 (+ 574us) client_negotiation.cc:769] Sending connection context
0619 00:52:44.801367 (+ 408us) client_negotiation.cc:240] Negotiation successful
0619 00:52:44.802047 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":394,"threads_started":1}
I20260619 00:52:44.805213 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.778806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35227 (local address 127.0.97.62:42755)
0619 00:52:44.779037 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:44.779044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:44.780604 (+ 1560us) server_negotiation.cc:407] Connection header received
0619 00:52:44.782982 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:44.782992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:44.783103 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:44.783227 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:44.785525 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.786496 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.789330 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.789942 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.790087 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.794425 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:44.794434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:44.794441 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:44.794488 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:44.798074 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.799041 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.799048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.799052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.799228 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.799753 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.799757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.799759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.800221 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:44.800376 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:44.804697 (+ 4321us) server_negotiation.cc:299] Negotiation successful
0619 00:52:44.804971 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:52:44.832259 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.812665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49857 (local address 127.0.97.62:42755)
0619 00:52:44.812882 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:44.812890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:44.813862 (+ 972us) server_negotiation.cc:407] Connection header received
0619 00:52:44.814273 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:44.814279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:44.814372 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:44.814511 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:44.816572 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.817757 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.820031 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.820929 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.821080 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.825916 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:44.825923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:44.825926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:44.825965 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:44.828725 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.829807 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.829812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.829814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.829945 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:44.830402 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:44.830406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:44.830408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:44.830739 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:44.830844 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:44.831633 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 00:52:44.831918 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:52:44.832554 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:44.813004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.2:49857)
0619 00:52:44.813754 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:44.813777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:44.813936 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:44.814722 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:44.814730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:44.815585 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:44.816324 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.816340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.817965 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.817973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.819834 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:44.819847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:44.821418 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:44.821428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:44.821702 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:44.822805 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:44.822844 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:44.825679 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:44.828920 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.828930 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.828937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.829665 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.830076 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:44.830084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:44.830090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:44.830282 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:44.830867 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:44.830874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:44.831146 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 00:52:44.831355 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 00:52:44.831990 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":446,"threads_started":1}
I20260619 00:52:44.834267 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46109
I20260619 00:52:44.834309 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46109 every 8 connection(s)
I20260619 00:52:44.835565 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:52:44.837908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1725
I20260619 00:52:44.838073 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260619 00:52:44.838109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:52:44.838318 1860 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:44.838711 388 external_mini_cluster.cc:1428] Setting key a32b1a46677a5f37dde2f77c82e167ac
I20260619 00:52:44.839155 1860 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260619 00:52:44.840725 1513 ts_manager.cc:194] Registered new tserver with Master: 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109)
I20260619 00:52:44.841645 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35227
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:52:44.877651 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:45.202313 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:45.202625 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:45.202793 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:45.210171 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:45.210283 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:45.210446 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:45.210561 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:52:45.220217 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:45.221570 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:45.223064 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:45.236238 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:45.236258 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:45.236562 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:45.237051 1864 server_base.cc:1034] running on GCE node
I20260619 00:52:45.237676 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:45.238608 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:45.239825 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781830365239766 us; error 33 us; skew 500 ppm
I20260619 00:52:45.240157 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:45 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:45.243603 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:52:45.245219 1864 webserver.cc:460] Webserver started at http://127.0.97.3:42333/ using document root <none> and password file <none>
I20260619 00:52:45.245934 1864 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:45.246050 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:45.246439 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:45.249657 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
format_stamp: "Formatted at 2026-06-19 00:52:45 on dist-test-slave-r7bh"
server_key: "8a07020190bb10af15eb2aa2e28139da"
server_key_iv: "68329b67185347ad914afe5db59a1f18"
server_key_version: "encryptionkey@0"
I20260619 00:52:45.250437 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
format_stamp: "Formatted at 2026-06-19 00:52:45 on dist-test-slave-r7bh"
server_key: "8a07020190bb10af15eb2aa2e28139da"
server_key_iv: "68329b67185347ad914afe5db59a1f18"
server_key_version: "encryptionkey@0"
I20260619 00:52:45.256057 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 00:52:45.260087 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:45.261585 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:45.261727 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
format_stamp: "Formatted at 2026-06-19 00:52:45 on dist-test-slave-r7bh"
server_key: "8a07020190bb10af15eb2aa2e28139da"
server_key_iv: "68329b67185347ad914afe5db59a1f18"
server_key_version: "encryptionkey@0"
I20260619 00:52:45.261955 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:45.277493 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:45.281785 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:45.282027 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:45.284703 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:45.284830 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:45.285027 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:45.285135 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:52:45 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:45.308753 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.285312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38423 (local address 127.0.97.62:42755)
0619 00:52:45.285518 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.285526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.287030 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 00:52:45.289089 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.289094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.289185 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.289315 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:45.291531 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.292840 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.295525 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.296281 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.296475 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.301499 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:45.301509 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:45.301516 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:45.301568 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:45.305199 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.305977 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.305981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.305983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.306102 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.306568 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.306575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.306578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.306916 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:45.307086 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.307894 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.308227 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 00:52:45.309473 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.285932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.3:38423)
0619 00:52:45.286822 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.286862 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.288734 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.289586 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.289601 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.290442 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:45.291298 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.291321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.293009 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.293021 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.295329 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.295349 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.296681 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.296694 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.297069 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.297897 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:45.297939 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:45.301205 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:45.305360 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.305370 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.305395 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.305828 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.306212 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.306217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.306221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.306418 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.307063 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:45.307071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:45.307429 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.307740 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.308227 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":445,"threads_started":1}
I20260619 00:52:45.323320 1721 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260619 00:52:45.341691 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.324896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49719 (local address 127.0.97.62:42755)
0619 00:52:45.325303 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.325310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.325622 (+ 312us) server_negotiation.cc:407] Connection header received
0619 00:52:45.325881 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.325889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.325978 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.326083 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:45.327766 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.329057 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.331589 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.332046 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.332211 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.335083 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:45.335088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:45.335092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:45.335120 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:45.338287 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.339145 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.339153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.339158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.339325 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.339784 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.339792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.339796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.340114 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:45.340276 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.341008 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.341258 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260619 00:52:45.341939 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.324781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.3:49719)
0619 00:52:45.325520 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.325543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.325738 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.326180 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.326188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.326978 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:45.327585 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.327605 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.329246 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.329252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.331447 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.331460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.332336 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.332348 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.332585 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.333535 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:45.333566 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:45.334885 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:45.338418 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.338425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.338429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.339039 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.339479 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.339485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.339489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.339654 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.340261 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:45.340269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:45.340588 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.340768 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.341237 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":600,"threads_started":1}
I20260619 00:52:45.344580 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42113
I20260619 00:52:45.344592 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42113 every 8 connection(s)
I20260619 00:52:45.346095 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:52:45.348201 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260619 00:52:45.348436 1999 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:45.349148 1999 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260619 00:52:45.350436 1513 ts_manager.cc:194] Registered new tserver with Master: 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:45.351212 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38423
I20260619 00:52:45.355831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 1864
I20260619 00:52:45.355997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:52:45.356374 388 external_mini_cluster.cc:1428] Setting key a02d282bba913a853fc10088c8ab13f0
I20260619 00:52:45.359352 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:52:45 dist-test-slave-r7bh krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:52:45 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:45.404647 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.390841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.0.97.62:42755)
0619 00:52:45.391058 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.391063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.391143 (+ 80us) server_negotiation.cc:407] Connection header received
0619 00:52:45.391291 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.391295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.391357 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.391461 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:45.392902 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.393620 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.394924 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.395329 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.395443 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.399519 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:45.399524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:45.399528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:45.399564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:45.402110 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.402755 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.402759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.402762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.402893 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.403344 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.403348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.403351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.403637 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:45.403815 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.404095 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.404259 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:52:45.409189 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:45.415814 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:45.442188 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.428737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.97.1:33567)
0619 00:52:45.431136 (+ 2399us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.431145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.432669 (+ 1524us) server_negotiation.cc:407] Connection header received
0619 00:52:45.433272 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.433281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.433549 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.433715 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.434674 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.436211 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.439099 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.440604 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.440704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.441016 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.441687 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.441878 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2240,"thread_start_us":2114,"threads_started":1}
I20260619 00:52:45.443333 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.428519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.0.1:38506)
0619 00:52:45.432376 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.432532 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.432887 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.434013 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.434020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.434048 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.434477 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.434497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.436392 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.436397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.438910 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.438929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.441205 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.441211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.441414 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.441477 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.442719 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.442941 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3504,"thread_start_us":3366,"threads_started":1}
I20260619 00:52:45.444412 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.428684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46109 (local address 127.0.0.1:37684)
0619 00:52:45.432376 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.432442 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.432630 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.434606 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.434613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.434643 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.435043 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.435056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.437928 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.437934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.440182 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.440198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.443580 (+ 3382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.443586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.443816 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.443855 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.443957 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.444097 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3462,"thread_start_us":3288,"threads_started":1}
I20260619 00:52:45.444653 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.429518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.97.2:46109)
0619 00:52:45.431548 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.431557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.432647 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 00:52:45.433943 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.433950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.434207 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.434410 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.435223 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.437753 (+ 2530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.440413 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.443074 (+ 2661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.443200 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.443433 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.444208 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.444389 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":1741,"threads_started":1}
I20260619 00:52:45.448231 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 (DEFAULT_TABLE table=test-table [id=131ee49b452546458d28f72e5e2c3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.448231 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 (DEFAULT_TABLE table=test-table [id=131ee49b452546458d28f72e5e2c3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.449760 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1388e4b88b1e491a8d898f821ecbdd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.449775 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1388e4b88b1e491a8d898f821ecbdd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.451380 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.432841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42113 (local address 127.0.0.1:34284)
0619 00:52:45.443591 (+ 10750us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.443616 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.443775 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.445012 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.445018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.445048 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.445418 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.445429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.447315 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.447350 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.449247 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.449260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.450609 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.450616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.450784 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.450818 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.450902 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.451035 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10541,"thread_start_us":11036,"threads_started":1}
I20260619 00:52:45.451426 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.433191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.97.3:42113)
0619 00:52:45.444204 (+ 11013us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.444213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.444247 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:52:45.444354 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.444362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.444656 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.444908 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.445622 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.447189 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.449414 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.450411 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.450562 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.450800 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.450962 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.451104 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10844,"thread_start_us":10687,"threads_started":1}
I20260619 00:52:45.460521 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 (DEFAULT_TABLE table=test-table [id=131ee49b452546458d28f72e5e2c3b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.461935 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1388e4b88b1e491a8d898f821ecbdd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.467386 2019 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Bootstrap starting.
I20260619 00:52:45.467733 2018 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Bootstrap starting.
I20260619 00:52:45.471385 2018 tablet_bootstrap.cc:654] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.471477 2019 tablet_bootstrap.cc:654] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.472477 2018 log.cc:824] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:45.472762 2019 log.cc:824] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:45.475741 2018 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: No bootstrap required, opened a new log
I20260619 00:52:45.476048 2018 ts_tablet_manager.cc:1397] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 00:52:45.478387 2022 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Bootstrap starting.
I20260619 00:52:45.479672 2019 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: No bootstrap required, opened a new log
I20260619 00:52:45.480047 2019 ts_tablet_manager.cc:1397] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 00:52:45.482267 2022 tablet_bootstrap.cc:654] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.483556 2022 log.cc:824] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:45.484442 2018 raft_consensus.cc:348] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.485134 2018 raft_consensus.cc:374] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.485270 2018 raft_consensus.cc:729] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6befbc93174f3393b1ae3a85b84324, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.486109 2018 consensus_queue.cc:260] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.486799 2022 tablet_bootstrap.cc:492] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: No bootstrap required, opened a new log
I20260619 00:52:45.487123 2022 ts_tablet_manager.cc:1397] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 00:52:45.487187 1860 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260619 00:52:45.487871 2018 ts_tablet_manager.cc:1428] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260619 00:52:45.490653 2019 raft_consensus.cc:348] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.491012 2019 raft_consensus.cc:374] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.491146 2019 raft_consensus.cc:729] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edc4d49688c41659b9fce7b52d2b3e2, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.491760 2019 consensus_queue.cc:260] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.493562 2019 ts_tablet_manager.cc:1428] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 00:52:45.496681 2022 raft_consensus.cc:348] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.497108 2022 raft_consensus.cc:374] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.497188 2022 raft_consensus.cc:729] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a351cbcbdb4e978ef4ee7319bce9d0, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.497632 2022 consensus_queue.cc:260] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.498728 1999 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260619 00:52:45.499253 2022 ts_tablet_manager.cc:1428] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
W20260619 00:52:45.564989 1722 tablet.cc:2369] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:45.566387 2024 raft_consensus.cc:482] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:45.566766 2024 raft_consensus.cc:504] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.568332 2024 leader_election.cc:288] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1:33567), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:45.577500 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.570250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35311 (local address 127.0.97.3:42113)
0619 00:52:45.570458 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.570464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.570896 (+ 432us) server_negotiation.cc:407] Connection header received
0619 00:52:45.571029 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.571032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.571093 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.571223 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.571657 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.572642 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.574981 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.576191 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.576315 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.576422 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.577050 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.577202 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:52:45.577500 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.569423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40987 (local address 127.0.97.1:33567)
0619 00:52:45.569676 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.569685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.570108 (+ 423us) server_negotiation.cc:407] Connection header received
0619 00:52:45.570343 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.570352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.570462 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.570572 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.571207 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.572288 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.574873 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.576146 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.576314 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.576422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.577050 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.577202 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:52:45.577677 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.569252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.97.2:40987)
0619 00:52:45.569958 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.569975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.570164 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.570606 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.570611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.570638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.571061 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.571070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.572424 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.572428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.574686 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.574703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.576524 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.576529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.576784 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.576878 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.576998 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.577208 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":67,"thread_start_us":396,"threads_started":1}
I20260619 00:52:45.578037 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.570107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42113 (local address 127.0.97.2:35311)
0619 00:52:45.570750 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.570771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.570892 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.571247 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.571250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.571268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.571517 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.571524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.572759 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.572764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.574820 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.574832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.576513 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.576520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.576779 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.576882 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.576990 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.577208 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":122,"thread_start_us":291,"threads_started":1}
I20260619 00:52:45.578933 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1388e4b88b1e491a8d898f821ecbdd97" candidate_uuid: "7a6befbc93174f3393b1ae3a85b84324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" is_pre_election: true
I20260619 00:52:45.579066 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1388e4b88b1e491a8d898f821ecbdd97" candidate_uuid: "7a6befbc93174f3393b1ae3a85b84324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" is_pre_election: true
I20260619 00:52:45.579469 1676 raft_consensus.cc:2454] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6befbc93174f3393b1ae3a85b84324 in term 0.
I20260619 00:52:45.579587 1949 raft_consensus.cc:2454] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6befbc93174f3393b1ae3a85b84324 in term 0.
I20260619 00:52:45.580116 1743 leader_election.cc:302] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6befbc93174f3393b1ae3a85b84324, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:45.580543 2024 raft_consensus.cc:2781] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:45.580714 2024 raft_consensus.cc:482] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:45.580876 2024 raft_consensus.cc:3037] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.583524 2024 raft_consensus.cc:504] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.584091 2024 leader_election.cc:288] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [CANDIDATE]: Term 1 election: Requested vote from peers 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1:33567), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:45.584648 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1388e4b88b1e491a8d898f821ecbdd97" candidate_uuid: "7a6befbc93174f3393b1ae3a85b84324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edc4d49688c41659b9fce7b52d2b3e2"
I20260619 00:52:45.584864 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1388e4b88b1e491a8d898f821ecbdd97" candidate_uuid: "7a6befbc93174f3393b1ae3a85b84324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
I20260619 00:52:45.584906 1676 raft_consensus.cc:3037] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.585078 1949 raft_consensus.cc:3037] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.587255 1676 raft_consensus.cc:2454] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6befbc93174f3393b1ae3a85b84324 in term 1.
I20260619 00:52:45.587495 1949 raft_consensus.cc:2454] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6befbc93174f3393b1ae3a85b84324 in term 1.
I20260619 00:52:45.587704 1743 leader_election.cc:302] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6befbc93174f3393b1ae3a85b84324, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:45.588040 2024 raft_consensus.cc:2781] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Leader election won for term 1
W20260619 00:52:45.588178 1861 tablet.cc:2369] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:45.588577 2024 raft_consensus.cc:686] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 LEADER]: Becoming Leader. State: Replica: 7a6befbc93174f3393b1ae3a85b84324, State: Running, Role: LEADER
I20260619 00:52:45.589191 2024 consensus_queue.cc:237] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.593879 1513 catalog_manager.cc:5495] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7a6befbc93174f3393b1ae3a85b84324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } health_report { overall_health: UNKNOWN } } }
W20260619 00:52:45.597679 2000 tablet.cc:2369] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:45.645078 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.639413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.0.97.2:46109)
0619 00:52:45.639594 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.639598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.639665 (+ 67us) server_negotiation.cc:407] Connection header received
0619 00:52:45.639983 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.639987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.640062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.640187 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:45.640674 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.641587 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.642956 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.643395 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.643525 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.643983 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:45.644505 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:45.644619 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.644726 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.644862 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:52:45.652884 1676 raft_consensus.cc:1261] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Refusing update from remote peer 7a6befbc93174f3393b1ae3a85b84324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:45.652909 1949 raft_consensus.cc:1261] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Refusing update from remote peer 7a6befbc93174f3393b1ae3a85b84324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:45.653735 2024 consensus_queue.cc:1035] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:45.654155 2029 consensus_queue.cc:1035] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:45.686569 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:45.686875 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:45.690752 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Sending DeleteTablet for 3 replicas of tablet 1388e4b88b1e491a8d898f821ecbdd97
I20260619 00:52:45.691963 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37684
I20260619 00:52:45.692067 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34284
I20260619 00:52:45.693440 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 1388e4b88b1e491a8d898f821ecbdd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38506
I20260619 00:52:45.695642 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:45.696462 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:45.701557 2043 tablet_replica.cc:330] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: stopping tablet replica
I20260619 00:52:45.702186 2043 raft_consensus.cc:2229] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:45.702445 2042 tablet_replica.cc:330] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: stopping tablet replica
I20260619 00:52:45.702837 2043 raft_consensus.cc:2258] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:45.702912 2042 raft_consensus.cc:2229] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:45.703430 2042 raft_consensus.cc:2258] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:45.705964 2042 ts_tablet_manager.cc:1905] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:45.707415 2041 tablet_replica.cc:330] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: stopping tablet replica
I20260619 00:52:45.707854 2041 raft_consensus.cc:2229] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:45.708596 2041 raft_consensus.cc:2258] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:45.713860 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 719addc7e56d424f88ab29650bff2b6b (DEFAULT_TABLE table=test-table [id=677c988e5d54436fafa7d4bdcde5678f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.714087 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 719addc7e56d424f88ab29650bff2b6b (DEFAULT_TABLE table=test-table [id=677c988e5d54436fafa7d4bdcde5678f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.714632 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719addc7e56d424f88ab29650bff2b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.714834 2043 ts_tablet_manager.cc:1905] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:45.715005 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719addc7e56d424f88ab29650bff2b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.717687 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 719addc7e56d424f88ab29650bff2b6b (DEFAULT_TABLE table=test-table [id=677c988e5d54436fafa7d4bdcde5678f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:45.718127 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719addc7e56d424f88ab29650bff2b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.721475 2018 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Bootstrap starting.
I20260619 00:52:45.722077 2043 ts_tablet_manager.cc:1918] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:52:45.722261 2043 log.cc:1196] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/1388e4b88b1e491a8d898f821ecbdd97
I20260619 00:52:45.723028 2018 tablet_bootstrap.cc:654] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.723071 2043 ts_tablet_manager.cc:1939] T 1388e4b88b1e491a8d898f821ecbdd97 P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting consensus metadata
I20260619 00:52:45.725006 1500 catalog_manager.cc:4841] TS 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1:33567): tablet 1388e4b88b1e491a8d898f821ecbdd97 (table test-table [id=131ee49b452546458d28f72e5e2c3b38]) successfully deleted
I20260619 00:52:45.730064 2019 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Bootstrap starting.
I20260619 00:52:45.731705 2019 tablet_bootstrap.cc:654] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.734436 2018 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: No bootstrap required, opened a new log
I20260619 00:52:45.734651 2018 ts_tablet_manager.cc:1397] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260619 00:52:45.735908 2018 raft_consensus.cc:348] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.736227 2018 raft_consensus.cc:374] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.736351 2018 raft_consensus.cc:729] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6befbc93174f3393b1ae3a85b84324, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.736601 2022 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Bootstrap starting.
I20260619 00:52:45.736712 2018 consensus_queue.cc:260] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.738255 2022 tablet_bootstrap.cc:654] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.738250 2018 ts_tablet_manager.cc:1428] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 00:52:45.743794 2022 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: No bootstrap required, opened a new log
I20260619 00:52:45.743974 2022 ts_tablet_manager.cc:1397] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 00:52:45.744628 2022 raft_consensus.cc:348] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.744747 2041 ts_tablet_manager.cc:1905] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:45.744824 2022 raft_consensus.cc:374] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.744953 2022 raft_consensus.cc:729] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a351cbcbdb4e978ef4ee7319bce9d0, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.745270 2022 consensus_queue.cc:260] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.745836 2022 ts_tablet_manager.cc:1428] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:45.748059 2019 tablet_bootstrap.cc:492] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: No bootstrap required, opened a new log
I20260619 00:52:45.748253 2019 ts_tablet_manager.cc:1397] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20260619 00:52:45.749503 2019 raft_consensus.cc:348] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.749788 2019 raft_consensus.cc:374] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.749953 2019 raft_consensus.cc:729] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edc4d49688c41659b9fce7b52d2b3e2, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.750280 2019 consensus_queue.cc:260] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.750777 2042 ts_tablet_manager.cc:1918] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:52:45.750938 2042 log.cc:1196] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/1388e4b88b1e491a8d898f821ecbdd97
I20260619 00:52:45.751169 2019 ts_tablet_manager.cc:1428] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 00:52:45.751482 2042 ts_tablet_manager.cc:1939] T 1388e4b88b1e491a8d898f821ecbdd97 P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting consensus metadata
I20260619 00:52:45.753039 1500 catalog_manager.cc:4841] TS 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113): tablet 1388e4b88b1e491a8d898f821ecbdd97 (table test-table [id=131ee49b452546458d28f72e5e2c3b38]) successfully deleted
I20260619 00:52:45.759485 2041 ts_tablet_manager.cc:1918] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:52:45.759673 2041 log.cc:1196] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/1388e4b88b1e491a8d898f821ecbdd97
I20260619 00:52:45.760205 2041 ts_tablet_manager.cc:1939] T 1388e4b88b1e491a8d898f821ecbdd97 P 7a6befbc93174f3393b1ae3a85b84324: Deleting consensus metadata
I20260619 00:52:45.761997 1499 catalog_manager.cc:4841] TS 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109): tablet 1388e4b88b1e491a8d898f821ecbdd97 (table test-table [id=131ee49b452546458d28f72e5e2c3b38]) successfully deleted
I20260619 00:52:45.764534 2025 raft_consensus.cc:482] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:45.764694 2025 raft_consensus.cc:504] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.766710 2025 leader_election.cc:288] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:45.774778 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.767448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60603 (local address 127.0.97.3:42113)
0619 00:52:45.767613 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.767620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.768638 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 00:52:45.768801 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.768821 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.768900 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.768994 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.769588 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.770664 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.773107 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.773846 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.774009 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.774109 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.774390 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.774520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 00:52:45.774849 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.767075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42113 (local address 127.0.97.1:60603)
0619 00:52:45.768524 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.768544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.768662 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.769020 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.769026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.769058 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.769425 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.769439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.770883 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.770889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.772917 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.772929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.774065 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.774071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.774237 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.774261 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.774365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.774500 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":1122,"threads_started":1}
I20260619 00:52:45.775419 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719addc7e56d424f88ab29650bff2b6b" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" is_pre_election: true
I20260619 00:52:45.775663 1949 raft_consensus.cc:2454] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 0.
I20260619 00:52:45.776186 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.766825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46109 (local address 127.0.97.1:40461)
0619 00:52:45.767807 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.767823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.767964 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.768465 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.768470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.768495 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:45.768891 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.768908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.770189 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.770195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.772877 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.772891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.774744 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.774751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.775536 (+ 785us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.775571 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.775701 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.775832 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":742,"threads_started":1}
I20260619 00:52:45.776214 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.767146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40461 (local address 127.0.97.2:46109)
0619 00:52:45.767370 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.767379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.767910 (+ 531us) server_negotiation.cc:407] Connection header received
0619 00:52:45.768106 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.768114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.768208 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.768296 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:45.769094 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.770058 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.773055 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.774337 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.774496 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.774588 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.775740 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.775883 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:52:45.776217 1603 leader_election.cc:302] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a351cbcbdb4e978ef4ee7319bce9d0, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:45.776629 2025 raft_consensus.cc:2781] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:45.776716 2025 raft_consensus.cc:482] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:45.776775 2025 raft_consensus.cc:3037] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.777220 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719addc7e56d424f88ab29650bff2b6b" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6befbc93174f3393b1ae3a85b84324" is_pre_election: true
I20260619 00:52:45.777678 1810 raft_consensus.cc:2454] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 0.
I20260619 00:52:45.780249 2025 raft_consensus.cc:504] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.780938 2025 leader_election.cc:288] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:45.781237 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719addc7e56d424f88ab29650bff2b6b" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6befbc93174f3393b1ae3a85b84324"
I20260619 00:52:45.781445 1810 raft_consensus.cc:3037] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.781637 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719addc7e56d424f88ab29650bff2b6b" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
I20260619 00:52:45.781831 1949 raft_consensus.cc:3037] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:45.783727 1949 raft_consensus.cc:2454] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 1.
I20260619 00:52:45.784112 1603 leader_election.cc:302] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a351cbcbdb4e978ef4ee7319bce9d0, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:45.784389 2025 raft_consensus.cc:2781] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:45.784703 1810 raft_consensus.cc:2454] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 1.
I20260619 00:52:45.784999 2025 raft_consensus.cc:686] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9edc4d49688c41659b9fce7b52d2b3e2, State: Running, Role: LEADER
I20260619 00:52:45.785427 2025 consensus_queue.cc:237] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.789062 1514 catalog_manager.cc:5495] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:52:45 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:45.841565 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.823214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.97.62:42755)
0619 00:52:45.823376 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:45.823383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:45.823980 (+ 597us) server_negotiation.cc:407] Connection header received
0619 00:52:45.824172 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:45.824181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:45.824268 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:45.824374 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:45.826015 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.826714 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.828108 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.828508 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.828627 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.835836 (+ 7209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:45.835841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:45.835846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:45.835884 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:45.838797 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.839464 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.839471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.839476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.839601 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:45.839983 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:45.839989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:45.839995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:45.840299 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:45.840446 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:45.840877 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 00:52:45.841253 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":55}
I20260619 00:52:45.841573 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:45.823303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.0.1:54642)
0619 00:52:45.823871 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:45.823886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:45.824056 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:45.824392 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:45.824396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:45.825603 (+ 1207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:45.825864 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.825873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.826835 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.826842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.827978 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:45.827989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:45.828680 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:45.828685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:45.828902 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:45.833321 (+ 4419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:45.833344 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:45.835668 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:45.838944 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.838956 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.838964 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.839345 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.839705 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:45.839711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:45.839715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:45.839856 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:45.840425 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:45.840433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:45.840590 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 00:52:45.840755 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 00:52:45.841096 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":338,"threads_started":1}
I20260619 00:52:45.850205 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54642:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 00:52:45.851485 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:45.859586 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 6421b45b27d94836a090d930f7b14a43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d82427d984a42c38c6aed5a9241def5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:45.859572 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 6421b45b27d94836a090d930f7b14a43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d82427d984a42c38c6aed5a9241def5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:45.860231 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6421b45b27d94836a090d930f7b14a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.860234 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6421b45b27d94836a090d930f7b14a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.866253 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 6421b45b27d94836a090d930f7b14a43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d82427d984a42c38c6aed5a9241def5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:45.866849 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6421b45b27d94836a090d930f7b14a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:45.868896 2019 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: Bootstrap starting.
I20260619 00:52:45.869316 2018 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: Bootstrap starting.
I20260619 00:52:45.871132 2019 tablet_bootstrap.cc:654] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.872233 2018 tablet_bootstrap.cc:654] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.874457 2022 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: Bootstrap starting.
I20260619 00:52:45.876555 2022 tablet_bootstrap.cc:654] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:45.887235 2022 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: No bootstrap required, opened a new log
I20260619 00:52:45.887513 2022 ts_tablet_manager.cc:1397] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260619 00:52:45.888336 2022 raft_consensus.cc:348] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.888662 2022 raft_consensus.cc:374] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.888741 2022 raft_consensus.cc:729] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a351cbcbdb4e978ef4ee7319bce9d0, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.889120 2022 consensus_queue.cc:260] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.889847 2022 ts_tablet_manager.cc:1428] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:45.889710 2026 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:45.890056 2026 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:45.891902 2019 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: No bootstrap required, opened a new log
I20260619 00:52:45.892134 2019 ts_tablet_manager.cc:1397] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20260619 00:52:45.893074 2019 raft_consensus.cc:348] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.893373 2019 raft_consensus.cc:374] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.893471 2019 raft_consensus.cc:729] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edc4d49688c41659b9fce7b52d2b3e2, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.893944 2018 tablet_bootstrap.cc:492] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: No bootstrap required, opened a new log
I20260619 00:52:45.893790 2019 consensus_queue.cc:260] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.894150 2018 ts_tablet_manager.cc:1397] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.010s
I20260619 00:52:45.894529 2019 ts_tablet_manager.cc:1428] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 00:52:45.894400 2025 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:45.894742 2025 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:45.895063 2018 raft_consensus.cc:348] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.895321 2018 raft_consensus.cc:374] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:45.895432 2018 raft_consensus.cc:729] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6befbc93174f3393b1ae3a85b84324, State: Initialized, Role: FOLLOWER
I20260619 00:52:45.895777 2018 consensus_queue.cc:260] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:45.896488 2018 ts_tablet_manager.cc:1428] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 00:52:45.896287 2029 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:45.896642 2029 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:46.092938 2025 raft_consensus.cc:482] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:46.093199 2025 raft_consensus.cc:504] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:46.093981 2025 leader_election.cc:288] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:46.094542 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6421b45b27d94836a090d930f7b14a43" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" is_pre_election: true
I20260619 00:52:46.094592 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6421b45b27d94836a090d930f7b14a43" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6befbc93174f3393b1ae3a85b84324" is_pre_election: true
I20260619 00:52:46.094816 1949 raft_consensus.cc:2454] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 0.
I20260619 00:52:46.094851 1810 raft_consensus.cc:2454] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 0.
I20260619 00:52:46.095440 1602 leader_election.cc:302] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a6befbc93174f3393b1ae3a85b84324, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:46.095755 2025 raft_consensus.cc:2781] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:46.095867 2025 raft_consensus.cc:482] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:46.095996 2025 raft_consensus.cc:3037] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:46.099839 2025 raft_consensus.cc:504] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:46.100644 2025 leader_election.cc:288] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109), 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113)
I20260619 00:52:46.101114 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6421b45b27d94836a090d930f7b14a43" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0"
I20260619 00:52:46.101126 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6421b45b27d94836a090d930f7b14a43" candidate_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6befbc93174f3393b1ae3a85b84324"
I20260619 00:52:46.101361 1949 raft_consensus.cc:3037] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:46.101370 1810 raft_consensus.cc:3037] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:46.104324 1949 raft_consensus.cc:2454] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 1.
I20260619 00:52:46.104487 1810 raft_consensus.cc:2454] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edc4d49688c41659b9fce7b52d2b3e2 in term 1.
I20260619 00:52:46.104897 1603 leader_election.cc:302] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a351cbcbdb4e978ef4ee7319bce9d0, 9edc4d49688c41659b9fce7b52d2b3e2; no voters:
I20260619 00:52:46.105187 2025 raft_consensus.cc:2781] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:46.105347 2025 raft_consensus.cc:686] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9edc4d49688c41659b9fce7b52d2b3e2, State: Running, Role: LEADER
I20260619 00:52:46.105588 2025 consensus_queue.cc:237] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } }
I20260619 00:52:46.107261 2049 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: TxnStatusTablet state changed. Reason: New leader 9edc4d49688c41659b9fce7b52d2b3e2. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.107600 2049 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:46.108174 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 00:52:46.108345 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 00:52:46.109529 1514 catalog_manager.cc:5495] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } health_report { overall_health: UNKNOWN } } }
I20260619 00:52:46.171829 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.164975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.0.1:38512)
0619 00:52:46.165304 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:46.165326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:46.165469 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:46.166672 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:46.166677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:46.166690 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:46.166898 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:46.166911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.168114 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.168120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:46.169312 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:46.169332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.169998 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.170004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:46.170195 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.170246 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:46.171232 (+ 986us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:46.171252 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 00:52:46.171325 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 00:52:46.171469 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 00:52:46.172135 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.165256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.97.1:33567)
0619 00:52:46.166213 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.166217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.166236 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:52:46.166349 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.166353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.166462 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.166639 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:46.167080 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.167954 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.169451 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.169853 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.169985 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.170491 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:46.171118 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:46.171283 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.171342 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.171418 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":650,"threads_started":1}
I20260619 00:52:46.176417 1949 raft_consensus.cc:1261] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Refusing update from remote peer 9edc4d49688c41659b9fce7b52d2b3e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:46.176594 1810 raft_consensus.cc:1261] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Refusing update from remote peer 9edc4d49688c41659b9fce7b52d2b3e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:46.177243 2049 consensus_queue.cc:1035] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:46.177587 2025 consensus_queue.cc:1035] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:46.199997 2026 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: TxnStatusTablet state changed. Reason: New leader 9edc4d49688c41659b9fce7b52d2b3e2. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.203785 2026 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:46.204268 2029 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: TxnStatusTablet state changed. Reason: New leader 9edc4d49688c41659b9fce7b52d2b3e2. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.204568 2029 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:46.206243 2065 mvcc.cc:205] Tried to move back new op lower bound from 7298377179851317248 to 7298377179571011584. Current Snapshot: MvccSnapshot[applied={T|T < 7298377179851317248}]
I20260619 00:52:46.209619 2049 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.209976 2049 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:46.217707 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298377179851317248 to 7298377179571011584. Current Snapshot: MvccSnapshot[applied={T|T < 7298377179851317248}]
I20260619 00:52:46.218211 2025 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.218684 2025 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 9edc4d49688c41659b9fce7b52d2b3e2: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:46.223644 2026 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.223917 2026 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 27a351cbcbdb4e978ef4ee7319bce9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:46.233229 2029 tablet_replica.cc:439] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edc4d49688c41659b9fce7b52d2b3e2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33567 } } peers { permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 } } peers { permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 } } }
I20260619 00:52:46.233662 2029 tablet_replica.cc:442] T 6421b45b27d94836a090d930f7b14a43 P 7a6befbc93174f3393b1ae3a85b84324: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:46.249538 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.237977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.97.1:33567)
0619 00:52:46.238166 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.238175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.244146 (+ 5971us) server_negotiation.cc:407] Connection header received
0619 00:52:46.244286 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.244293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.244402 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.244511 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:46.245081 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.246222 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.247720 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.248160 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.248329 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.248730 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:46.248879 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:46.248995 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.249143 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.249227 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:52:46.256076 2049 consensus_queue.cc:1035] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6befbc93174f3393b1ae3a85b84324" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:46.262200 2049 consensus_queue.cc:1035] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a351cbcbdb4e978ef4ee7319bce9d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:46.288280 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.271177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53039 (local address 127.0.97.1:33567)
0619 00:52:46.271345 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.271351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.282861 (+ 11510us) server_negotiation.cc:407] Connection header received
0619 00:52:46.283064 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.283069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.283162 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.283296 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:46.283665 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.284863 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.286436 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.286871 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.287020 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.287450 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:46.287592 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:46.287688 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.287826 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.287900 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":59}
I20260619 00:52:46.288363 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.270689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33567 (local address 127.0.97.1:53039)
0619 00:52:46.282709 (+ 12020us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:46.282731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:46.282902 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:46.283310 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:46.283313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:46.283323 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:46.283545 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:46.283552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.285010 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.285015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:46.286263 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:46.286275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.287074 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.287082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:46.287203 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.287259 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:46.287685 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:46.287694 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 00:52:46.287809 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 00:52:46.287947 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11830,"mutex_wait_us":3,"thread_start_us":12294,"threads_started":1}
I20260619 00:52:46.373955 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:46.374190 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:46.377254 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51162d3d3461419d8a56c19129ac9cd6: Sending DeleteTablet for 3 replicas of tablet 719addc7e56d424f88ab29650bff2b6b
I20260619 00:52:46.378104 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 719addc7e56d424f88ab29650bff2b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38506
I20260619 00:52:46.378664 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 719addc7e56d424f88ab29650bff2b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34284
I20260619 00:52:46.379312 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 719addc7e56d424f88ab29650bff2b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37684
I20260619 00:52:46.388211 2083 tablet_replica.cc:330] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: stopping tablet replica
I20260619 00:52:46.388533 2083 raft_consensus.cc:2229] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:46.389693 2083 raft_consensus.cc:2258] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:46.391666 2083 ts_tablet_manager.cc:1905] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:46.397993 2085 tablet_replica.cc:330] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: stopping tablet replica
I20260619 00:52:46.398351 2085 raft_consensus.cc:2229] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:46.398654 2085 raft_consensus.cc:2258] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:46.399844 2085 ts_tablet_manager.cc:1905] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:46.407804 2082 tablet_replica.cc:330] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: stopping tablet replica
I20260619 00:52:46.410686 2083 ts_tablet_manager.cc:1918] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:46.410915 2083 log.cc:1196] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/719addc7e56d424f88ab29650bff2b6b
I20260619 00:52:46.411126 2082 raft_consensus.cc:2229] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:46.411446 2083 ts_tablet_manager.cc:1939] T 719addc7e56d424f88ab29650bff2b6b P 27a351cbcbdb4e978ef4ee7319bce9d0: Deleting consensus metadata
I20260619 00:52:46.411880 2082 raft_consensus.cc:2258] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:46.412940 1500 catalog_manager.cc:4841] TS 27a351cbcbdb4e978ef4ee7319bce9d0 (127.0.97.3:42113): tablet 719addc7e56d424f88ab29650bff2b6b (table test-table [id=677c988e5d54436fafa7d4bdcde5678f]) successfully deleted
I20260619 00:52:46.413067 2082 ts_tablet_manager.cc:1905] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:46.420794 2082 ts_tablet_manager.cc:1918] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:46.421109 2082 log.cc:1196] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/719addc7e56d424f88ab29650bff2b6b
I20260619 00:52:46.421661 2082 ts_tablet_manager.cc:1939] T 719addc7e56d424f88ab29650bff2b6b P 9edc4d49688c41659b9fce7b52d2b3e2: Deleting consensus metadata
I20260619 00:52:46.422864 1500 catalog_manager.cc:4841] TS 9edc4d49688c41659b9fce7b52d2b3e2 (127.0.97.1:33567): tablet 719addc7e56d424f88ab29650bff2b6b (table test-table [id=677c988e5d54436fafa7d4bdcde5678f]) successfully deleted
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:46.425323 2085 ts_tablet_manager.cc:1918] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:46.425455 2085 log.cc:1196] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/719addc7e56d424f88ab29650bff2b6b
I20260619 00:52:46.426270 2085 ts_tablet_manager.cc:1939] T 719addc7e56d424f88ab29650bff2b6b P 7a6befbc93174f3393b1ae3a85b84324: Deleting consensus metadata
I20260619 00:52:46.427809 1499 catalog_manager.cc:4841] TS 7a6befbc93174f3393b1ae3a85b84324 (127.0.97.2:46109): tablet 719addc7e56d424f88ab29650bff2b6b (table test-table [id=677c988e5d54436fafa7d4bdcde5678f]) successfully deleted
I20260619 00:52:46.432788 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.402711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.0.97.1:33567)
0619 00:52:46.402912 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.402922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.413830 (+ 10908us) server_negotiation.cc:407] Connection header received
0619 00:52:46.414061 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.414067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.414168 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.414300 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:46.415748 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.416535 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.418829 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.419564 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.419658 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.424884 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:46.424897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:46.424917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:46.424957 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:46.429129 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.429864 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.429874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.429878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.430502 (+ 624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.430840 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.430845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.430848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.431348 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:46.431703 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.432124 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.432363 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 00:52:46.433580 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38538
I20260619 00:52:46.452178 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.436932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.0.97.62:42755)
0619 00:52:46.437848 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.437854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.437871 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:52:46.437924 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.437930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.437998 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.438102 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:46.439476 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.440666 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.442304 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.442992 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.443121 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.445409 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:46.445416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:46.445421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:46.445472 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:46.449185 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.449824 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.449830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.449834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.449974 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.450313 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.450318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.450322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.451078 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:46.451324 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.451578 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.451793 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":595,"threads_started":1}
W20260619 00:52:46.452858 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54650
I20260619 00:52:46.455379 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1585
I20260619 00:52:46.469363 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.456323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.0.97.1:33567)
0619 00:52:46.456542 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.456548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.456780 (+ 232us) server_negotiation.cc:407] Connection header received
0619 00:52:46.457028 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.457032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.457098 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.457210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:46.458289 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.459031 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.460223 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.460697 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.460862 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.463178 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:46.463182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:46.463186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:46.463220 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:46.466368 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.467039 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.467045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.467049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.467194 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.467826 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.467838 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.467840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.468160 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:46.468456 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.468761 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.469002 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 00:52:46.469933 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38540
W20260619 00:52:46.470530 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:46.470711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1585
I20260619 00:52:46.486763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1725
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:52:46.501757 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.487540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.0.97.2:46109)
0619 00:52:46.488028 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.488033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.488049 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:52:46.488102 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.488106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.488163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.488252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:46.489413 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.490122 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.491351 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.491753 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.491853 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.495797 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:46.495812 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:46.495828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:46.495868 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:46.498771 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.499629 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.499640 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.499644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.499799 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.500188 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.500191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.500193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.500645 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:46.501036 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.501302 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.501497 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
W20260619 00:52:46.502435 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37702
W20260619 00:52:46.503139 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:46.503331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1725
I20260619 00:52:46.524214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1864
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:52:46.539083 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:46.525043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.97.3:42113)
0619 00:52:46.525560 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:46.525565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:46.525582 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:52:46.525634 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:46.525638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:46.525700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:46.525796 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:46.527000 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.527765 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.529168 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:46.529646 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:46.529755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:46.533822 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:46.533837 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:46.533853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:46.533890 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:46.536688 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.537412 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.537420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.537423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.537564 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:46.537898 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:46.537901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:46.537903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:46.538228 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:46.538474 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:46.538648 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 00:52:46.538818 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
W20260619 00:52:46.539712 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34298
W20260619 00:52:46.540336 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:46.540474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1864
I20260619 00:52:46.557696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 1482
I20260619 00:52:46.558974 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:46.674312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 1482
2026-06-19T00:52:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4722 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[2096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[2096](info): set up 2 sockets
Jun 19 00:52:46 dist-test-slave-r7bh krb5kdc[2096](info): commencing operation
krb5kdc: starting...
W20260619 00:52:47.785076 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:52:47 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:47.968924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:48.298401 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:48.298766 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:48.298928 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:48.306357 2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:52:48.306509 2109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 00:52:48.306607 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:48.306727 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:48.306813 2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:52:48.306902 2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:52:48.317233 2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:48.318589 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:48.319990 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:48.332221 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:48.332278 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:48.332346 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:48.333076 2109 server_base.cc:1034] running on GCE node
I20260619 00:52:48.333882 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:48.335386 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:48.336666 2109 hybrid_clock.cc:648] HybridClock initialized: now 1781830368336599 us; error 51 us; skew 500 ppm
I20260619 00:52:48.337080 2109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:48 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:48.340404 2109 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:52:48.341923 2109 webserver.cc:460] Webserver started at http://127.0.97.62:44751/ using document root <none> and password file <none>
I20260619 00:52:48.342620 2109 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:48.342705 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:48.342970 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:48.345904 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "814f83052ebe42dba4285b2be6ecf6f5"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "6a8e71dbc4f7d6b2607a3a4379891d9a"
server_key_iv: "06913487043a6d0445d5dd99b7b1dc57"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.346637 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "814f83052ebe42dba4285b2be6ecf6f5"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "6a8e71dbc4f7d6b2607a3a4379891d9a"
server_key_iv: "06913487043a6d0445d5dd99b7b1dc57"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.351522 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:52:48.354871 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:48.356210 2109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:52:48.356416 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "814f83052ebe42dba4285b2be6ecf6f5"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "6a8e71dbc4f7d6b2607a3a4379891d9a"
server_key_iv: "06913487043a6d0445d5dd99b7b1dc57"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.356618 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:48.379964 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:48.383976 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:48.384264 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:48.405494 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41367
I20260619 00:52:48.405498 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41367 every 8 connection(s)
I20260619 00:52:48.406865 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:52:48.407490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2109
I20260619 00:52:48.407723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:52:48.408546 388 external_mini_cluster.cc:1428] Setting key 40a45bf1eeddfc984a50106953a337b0
I20260619 00:52:48.411494 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:48.422211 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Bootstrap starting.
Jun 19 00:52:48 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:48.426136 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:48.427268 2186 log.cc:824] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:48.430629 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: No bootstrap required, opened a new log
I20260619 00:52:48.431408 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:48.410590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.0.97.62:41367)
0619 00:52:48.411343 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:48.411355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:48.411394 (+ 39us) server_negotiation.cc:407] Connection header received
0619 00:52:48.413154 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:48.413176 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:48.413562 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:48.413990 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:48.415406 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.416910 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.418203 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.418905 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.419136 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:48.423548 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:48.423557 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:48.423573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:48.423617 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:48.427024 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:48.427868 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:48.427883 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:48.427895 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:48.428156 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:48.428540 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:48.428548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:48.428551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:48.429169 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:48.429373 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:48.429835 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 00:52:48.430239 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":361,"threads_started":1}
I20260619 00:52:48.438587 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } }
I20260619 00:52:48.438969 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:48.439069 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814f83052ebe42dba4285b2be6ecf6f5, State: Initialized, Role: FOLLOWER
I20260619 00:52:48.439642 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } }
I20260619 00:52:48.439836 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:52:48.440008 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:52:48.440194 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:48.442507 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } }
I20260619 00:52:48.443037 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814f83052ebe42dba4285b2be6ecf6f5; no voters:
I20260619 00:52:48.443624 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:52:48.443768 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:48.444478 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [term 1 LEADER]: Becoming Leader. State: Replica: 814f83052ebe42dba4285b2be6ecf6f5, State: Running, Role: LEADER
I20260619 00:52:48.445082 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } }
I20260619 00:52:48.445895 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:52:48.448465 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814f83052ebe42dba4285b2be6ecf6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } } }
I20260619 00:52:48.448594 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814f83052ebe42dba4285b2be6ecf6f5. Latest consensus state: current_term: 1 leader_uuid: "814f83052ebe42dba4285b2be6ecf6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814f83052ebe42dba4285b2be6ecf6f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41367 } } }
I20260619 00:52:48.448892 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:48.448957 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:48.451438 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:52:48.455827 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:52:48.464848 2197 catalog_manager.cc:1347] Generated new cluster ID: 130245a168984a1ea732c2d174c54f0d
I20260619 00:52:48.464986 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:52:48.495792 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:52:48.497821 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:52:48.506798 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Generated new TSK 0
I20260619 00:52:48.507870 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:52:48.556907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41367
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:48.888278 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:48.888629 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:48.888855 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:48.896319 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:48.896500 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:48.896646 2212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:52:48.896732 2212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:48.896893 2212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:52:48.907092 2212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:48.908510 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:48.909955 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:48.923569 2218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:48.923578 2217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:48.923769 2220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:48.924351 2212 server_base.cc:1034] running on GCE node
I20260619 00:52:48.925062 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:48.926059 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:48.927315 2212 hybrid_clock.cc:648] HybridClock initialized: now 1781830368927250 us; error 45 us; skew 500 ppm
I20260619 00:52:48.927713 2212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:48 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:48.930769 2212 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:52:48.932267 2212 webserver.cc:460] Webserver started at http://127.0.97.1:44893/ using document root <none> and password file <none>
I20260619 00:52:48.933002 2212 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:48.933109 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:48.933370 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:48.936249 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "ccc9ae1782e2477e9a4f8860514c8913"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "891f5551b403848fbcad899dc4340c95"
server_key_iv: "a88cb714f86aba942706bad98027a653"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.937047 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc9ae1782e2477e9a4f8860514c8913"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "891f5551b403848fbcad899dc4340c95"
server_key_iv: "a88cb714f86aba942706bad98027a653"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.942245 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 00:52:48.945781 2227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:48.947441 2212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 00:52:48.947580 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "ccc9ae1782e2477e9a4f8860514c8913"
format_stamp: "Formatted at 2026-06-19 00:52:48 on dist-test-slave-r7bh"
server_key: "891f5551b403848fbcad899dc4340c95"
server_key_iv: "a88cb714f86aba942706bad98027a653"
server_key_version: "encryptionkey@0"
I20260619 00:52:48.947758 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:48.970983 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:48.975569 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:48.975826 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:48.978668 2212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:48.978806 2212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:48.978936 2212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:48.979033 2212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:52:48 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:49.008111 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:48.979598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42739 (local address 127.0.97.62:41367)
0619 00:52:48.980088 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:48.980092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:48.981596 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 00:52:48.984285 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:48.984293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:48.984398 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:48.984490 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:48.987553 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.988666 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.991486 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.992304 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.992478 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:48.998826 (+ 6348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:48.998835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:48.998845 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:48.998899 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:49.002961 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.004254 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.004265 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.004272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.004467 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.005158 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.005168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.005174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.005642 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:49.005912 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:49.007262 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0619 00:52:49.007663 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260619 00:52:49.010011 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:48.980485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.1:42739)
0619 00:52:48.981412 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:48.981456 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:48.983830 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:48.985140 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:48.985149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:48.986073 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:48.987346 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:48.987372 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.988900 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.988907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:48.991214 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:48.991229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:48.992617 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:48.992627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:48.993047 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:48.994079 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:48.994123 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:48.998354 (+ 4231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:49.003247 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.003258 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.003281 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.004054 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.004632 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.004639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.004644 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.004995 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.005999 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:49.006015 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:49.006699 (+ 684us) client_negotiation.cc:769] Sending connection context
0619 00:52:49.007169 (+ 470us) client_negotiation.cc:240] Negotiation successful
0619 00:52:49.007990 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":405,"threads_started":1}
I20260619 00:52:49.021556 2212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38719
I20260619 00:52:49.021665 2347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38719 every 8 connection(s)
I20260619 00:52:49.023157 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:52:49.025540 2348 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41367
I20260619 00:52:49.025807 2348 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:49.026602 2348 heartbeater.cc:511] Master 127.0.97.62:41367 requested a full tablet report, sending...
I20260619 00:52:49.027434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2212
I20260619 00:52:49.027661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:52:49.028537 388 external_mini_cluster.cc:1428] Setting key a3357f7b9e29aea59687a3b7ee1e26bf
I20260619 00:52:49.029175 2141 ts_manager.cc:194] Registered new tserver with Master: ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719)
I20260619 00:52:49.031515 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42739
I20260619 00:52:49.035928 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.019431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.1:46711)
0619 00:52:49.020203 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:49.020220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:49.020424 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:49.020797 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:49.020802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:49.021650 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:49.022063 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.022076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.023182 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.023191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.025019 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.025034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.026066 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.026074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.026293 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.027073 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:49.027104 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:49.028292 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:49.032748 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.032754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.032759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.033426 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.033928 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.033933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.033938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.034106 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.034858 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:49.034864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:49.035057 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 00:52:49.035215 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 00:52:49.035560 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":492,"threads_started":1}
I20260619 00:52:49.036098 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.019321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46711 (local address 127.0.97.62:41367)
0619 00:52:49.019524 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:49.019529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:49.020332 (+ 803us) server_negotiation.cc:407] Connection header received
0619 00:52:49.020583 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:49.020587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:49.020655 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:49.020768 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:49.022237 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.023027 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.025202 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.025842 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.026023 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.028500 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:49.028509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:49.028514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:49.028552 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:49.032594 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.033585 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.033594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.033601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.033783 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.034271 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.034288 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.034294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.034698 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:49.034885 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:49.035385 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 00:52:49.035679 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:52:49.072628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41367
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:49.413353 2352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:49.413661 2352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:49.413822 2352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:49.420951 2352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:49.421090 2352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:49.421231 2352 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:52:49.421316 2352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:49.421443 2352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:52:49.431097 2352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:49.432596 2352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:49.433979 2352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:49.447000 2358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:49.447125 2357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:49.447104 2360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:49.447909 2352 server_base.cc:1034] running on GCE node
I20260619 00:52:49.448518 2352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:49.449542 2352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:49.450763 2352 hybrid_clock.cc:648] HybridClock initialized: now 1781830369450702 us; error 56 us; skew 500 ppm
I20260619 00:52:49.451067 2352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:49 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:49.455013 2352 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:52:49.459604 2352 webserver.cc:460] Webserver started at http://127.0.97.2:35687/ using document root <none> and password file <none>
I20260619 00:52:49.460991 2352 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:49.461325 2352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:49.461746 2352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:49.466013 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "b2e303399dba40459a33885dc8871f14"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "97c4da23eb6c198d62a426ebb065cc57"
server_key_iv: "df6994f3eaa3b0124b045bc517e12474"
server_key_version: "encryptionkey@0"
I20260619 00:52:49.467299 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b2e303399dba40459a33885dc8871f14"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "97c4da23eb6c198d62a426ebb065cc57"
server_key_iv: "df6994f3eaa3b0124b045bc517e12474"
server_key_version: "encryptionkey@0"
I20260619 00:52:49.474705 2352 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260619 00:52:49.480026 2367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:49.481997 2352 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 00:52:49.482197 2352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "b2e303399dba40459a33885dc8871f14"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "97c4da23eb6c198d62a426ebb065cc57"
server_key_iv: "df6994f3eaa3b0124b045bc517e12474"
server_key_version: "encryptionkey@0"
I20260619 00:52:49.482424 2352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:49.514046 2352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:49.519021 2352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:49.519346 2352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:49.522163 2352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:49.522312 2352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:49.522437 2352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:49.522558 2352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:52:49 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:49.547984 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.522940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59109 (local address 127.0.97.62:41367)
0619 00:52:49.523155 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:49.523162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:49.524691 (+ 1529us) server_negotiation.cc:407] Connection header received
0619 00:52:49.526977 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:49.526986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:49.527076 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:49.527200 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:49.529649 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.530727 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.533425 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.533934 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.534109 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.539921 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:49.539928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:49.539931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:49.539967 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:49.544128 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.545205 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.545211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.545216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.545418 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.546089 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.546093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.546095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.546387 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:49.546548 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:49.547401 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 00:52:49.547627 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:52:49.549134 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.523578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.2:59109)
0619 00:52:49.524485 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:49.524538 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:49.526647 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:49.527489 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:49.527500 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:49.528276 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:49.529395 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.529427 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.530893 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.530904 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.533255 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.533267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.534228 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.534238 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.534662 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.535592 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:49.535633 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:49.539588 (+ 3955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:49.544318 (+ 4730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.544326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.544341 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.545056 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.545564 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.545573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.545580 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.545949 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.546526 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:49.546535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:49.547029 (+ 494us) client_negotiation.cc:769] Sending connection context
0619 00:52:49.547303 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 00:52:49.547855 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":400,"threads_started":1}
I20260619 00:52:49.572595 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.556573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36489 (local address 127.0.97.62:41367)
0619 00:52:49.556805 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:49.556849 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:49.557476 (+ 627us) server_negotiation.cc:407] Connection header received
0619 00:52:49.557703 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:49.557709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:49.557799 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:49.557907 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:49.559350 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.560501 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.562686 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.563284 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.563444 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.566313 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:49.566322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:49.566328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:49.566372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:49.569483 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.570428 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.570434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.570438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.570601 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:49.571146 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:49.571149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:49.571152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:49.571504 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:49.571622 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:49.572198 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 00:52:49.572376 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:52:49.573226 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:49.556521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.2:36489)
0619 00:52:49.557342 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:49.557366 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:49.557544 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:49.557940 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:49.557945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:49.558621 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:49.559158 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.559173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.560661 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.560666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.562515 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:49.562544 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:49.563723 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:49.563728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:49.563854 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:49.564784 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:49.564865 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:49.566035 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:49.569614 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.569620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.569624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.570282 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.570735 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:49.570744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:49.570751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:49.570957 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:49.571629 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:49.571636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:49.571903 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 00:52:49.572092 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 00:52:49.572558 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":451,"threads_started":1}
I20260619 00:52:49.581040 2352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45887
I20260619 00:52:49.581130 2486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45887 every 8 connection(s)
I20260619 00:52:49.582365 2352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:52:49.584728 2487 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41367
I20260619 00:52:49.585039 2487 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:49.585721 2487 heartbeater.cc:511] Master 127.0.97.62:41367 requested a full tablet report, sending...
I20260619 00:52:49.587117 2141 ts_manager.cc:194] Registered new tserver with Master: b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887)
I20260619 00:52:49.588017 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59109
I20260619 00:52:49.592370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2352
I20260619 00:52:49.592553 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:52:49.593107 388 external_mini_cluster.cc:1428] Setting key bdeef009c14633a7488e0cc19a4fe67d
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:52:49.632241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41367
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:49.954440 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:49.954720 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:49.954905 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:49.962420 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:49.962548 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:49.962707 2491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:52:49.962787 2491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:52:49.962926 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:52:49.972965 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:49.974359 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:49.975752 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:49.988782 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:49.988868 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:49.988824 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:49.989846 2491 server_base.cc:1034] running on GCE node
I20260619 00:52:49.990506 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:49.991492 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:49.992717 2491 hybrid_clock.cc:648] HybridClock initialized: now 1781830369992669 us; error 48 us; skew 500 ppm
I20260619 00:52:49.993068 2491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:49 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:49.996222 2491 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:52:49.997880 2491 webserver.cc:460] Webserver started at http://127.0.97.3:34747/ using document root <none> and password file <none>
I20260619 00:52:49.998605 2491 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:49.998749 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:49.999055 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:50.002022 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "38cd251d4d404145bea2cab1232e054b"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "a72fa2ff8b6658fd525bfcc28a3d4a59"
server_key_iv: "1d7eb322fad39298cb430cf7f8853edb"
server_key_version: "encryptionkey@0"
I20260619 00:52:50.002841 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "38cd251d4d404145bea2cab1232e054b"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "a72fa2ff8b6658fd525bfcc28a3d4a59"
server_key_iv: "1d7eb322fad39298cb430cf7f8853edb"
server_key_version: "encryptionkey@0"
I20260619 00:52:50.008119 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 00:52:50.011952 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:50.013417 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:50.013604 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "38cd251d4d404145bea2cab1232e054b"
format_stamp: "Formatted at 2026-06-19 00:52:49 on dist-test-slave-r7bh"
server_key: "a72fa2ff8b6658fd525bfcc28a3d4a59"
server_key_iv: "1d7eb322fad39298cb430cf7f8853edb"
server_key_version: "encryptionkey@0"
I20260619 00:52:50.013825 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:50.033988 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:50.035784 2348 heartbeater.cc:503] Master 127.0.97.62:41367 was elected leader, sending a full tablet report...
I20260619 00:52:50.039325 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:50.039642 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:50.042443 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:50.042591 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:50.042711 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:50.042816 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:52:50 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:50.069073 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.043096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55301 (local address 127.0.97.62:41367)
0619 00:52:50.043413 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.043420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.044730 (+ 1310us) server_negotiation.cc:407] Connection header received
0619 00:52:50.047110 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.047115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.047224 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.047320 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:50.049709 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.050785 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.053165 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.053779 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.053947 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.060191 (+ 6244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:50.060197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:50.060201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:50.060237 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:50.064573 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.065740 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.065744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.065746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.065929 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.066701 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.066708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.066714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.067066 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:50.067259 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.068414 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.068668 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260619 00:52:50.070294 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.043730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.3:55301)
0619 00:52:50.044576 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.044613 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.046799 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.047650 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.047664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.048523 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:50.049513 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.049534 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.050962 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.050970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.053006 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.053019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.054082 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.054094 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.054563 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.055678 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:50.055713 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:50.059780 (+ 4067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:50.064787 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.064795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.064855 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.065553 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.066083 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.066090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.066096 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.066457 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.067346 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:50.067357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:50.067913 (+ 556us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.068281 (+ 368us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.068889 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":383,"threads_started":1}
I20260619 00:52:50.093775 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.076300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44813 (local address 127.0.97.62:41367)
0619 00:52:50.076532 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.076541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.077176 (+ 635us) server_negotiation.cc:407] Connection header received
0619 00:52:50.077450 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.077459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.077557 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.077696 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:50.078930 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.080031 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.082052 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.082495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.082656 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.085738 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:50.085747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:50.085755 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:50.085808 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:50.090224 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.091243 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.091252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.091258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.091450 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.092028 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.092035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.092039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.092486 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:50.092658 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.093176 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.093411 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:52:50.093870 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.076288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.97.3:44813)
0619 00:52:50.077063 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.077080 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.077263 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.077734 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.077744 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.078240 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:50.078709 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.078729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.080184 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.080190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.081895 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.081907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.082702 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.082708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.082980 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.083800 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:50.083827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:50.085442 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:50.090373 (+ 4931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.090378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.090382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.091016 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.091639 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.091644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.091648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.091860 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.092623 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:50.092634 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:50.092858 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.093029 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.093411 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":395,"threads_started":1}
I20260619 00:52:50.102322 2491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41717
I20260619 00:52:50.102366 2625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41717 every 8 connection(s)
I20260619 00:52:50.103631 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:52:50.105726 2626 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41367
I20260619 00:52:50.105973 2626 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:50.106598 2626 heartbeater.cc:511] Master 127.0.97.62:41367 requested a full tablet report, sending...
I20260619 00:52:50.107884 2141 ts_manager.cc:194] Registered new tserver with Master: 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:50.108709 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55301
I20260619 00:52:50.110662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2491
I20260619 00:52:50.110837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:52:50.111280 388 external_mini_cluster.cc:1428] Setting key 8d0588d5a14c72d77871d6e8a0176073
I20260619 00:52:50.114187 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:52:50 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:52:50 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:50.163646 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.149440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.0.97.62:41367)
0619 00:52:50.149672 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.149679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.149813 (+ 134us) server_negotiation.cc:407] Connection header received
0619 00:52:50.150045 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.150050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.150118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.150211 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:50.151833 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.152878 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.154164 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.154570 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.154666 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.158496 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:50.158501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:50.158504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:50.158539 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:50.161243 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.161853 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.161858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.161860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.162016 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.162386 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.162389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.162392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.162659 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:50.162837 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.163196 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.163367 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:52:50.167661 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:50.174387 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:50.199764 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.187670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41717 (local address 127.0.0.1:57528)
0619 00:52:50.189599 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.189622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.189757 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.192160 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.192167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.192196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.192676 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.192691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.194795 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.194802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.196989 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.197008 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.198878 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.198885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.199093 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.199147 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.199252 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.199437 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"thread_start_us":1008,"threads_started":1}
I20260619 00:52:50.200842 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.188458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.97.3:41717)
0619 00:52:50.191634 (+ 3176us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.191640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.191667 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:52:50.191746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.191751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.191940 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.192159 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.192948 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.194633 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.197157 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.198645 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.200024 (+ 1379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.200257 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.200328 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.200492 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3016,"thread_start_us":2861,"threads_started":1}
I20260619 00:52:50.200958 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.187321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45887 (local address 127.0.0.1:35400)
0619 00:52:50.188654 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.188706 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.188902 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.193552 (+ 4650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.193557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.193583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.193986 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.193997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.196085 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.196090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.198337 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.198353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.200092 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.200099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.200262 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.200301 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.200362 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.200525 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":3,"thread_start_us":1910,"threads_started":1}
I20260619 00:52:50.201354 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.190254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.97.1:38719)
0619 00:52:50.193216 (+ 2962us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.193227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.193917 (+ 690us) server_negotiation.cc:407] Connection header received
0619 00:52:50.193999 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.194004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.194146 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.194342 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.195283 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.196487 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.198598 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.199787 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.199948 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.200152 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.200717 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.200900 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2796,"thread_start_us":2663,"threads_started":1}
I20260619 00:52:50.201403 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.189666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.0.1:55396)
0619 00:52:50.193640 (+ 3974us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.193654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.193808 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.194568 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.194574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.194598 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.195084 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.195097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.196633 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.196639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.198386 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.198400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.200316 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.200325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.200527 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.200551 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.200666 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.200793 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3867,"mutex_wait_us":69,"thread_start_us":3733,"threads_started":1}
I20260619 00:52:50.201937 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.187445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.0.97.2:45887)
0619 00:52:50.192845 (+ 5400us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.192856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.192887 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:52:50.192976 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.192984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.193317 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.194308 (+ 991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.194496 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.195940 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.198530 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.199890 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.201208 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.201388 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.201448 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.201618 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5208,"thread_start_us":5072,"threads_started":1}
I20260619 00:52:50.204994 2557 tablet_service.cc:1467] Processing CreateTablet for tablet f483232bfc8f4b918c4442609ec1a05a (DEFAULT_TABLE table=test-table [id=ab95d2c8340b4c90a32d12e812408027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.205044 2417 tablet_service.cc:1467] Processing CreateTablet for tablet f483232bfc8f4b918c4442609ec1a05a (DEFAULT_TABLE table=test-table [id=ab95d2c8340b4c90a32d12e812408027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.205044 2278 tablet_service.cc:1467] Processing CreateTablet for tablet f483232bfc8f4b918c4442609ec1a05a (DEFAULT_TABLE table=test-table [id=ab95d2c8340b4c90a32d12e812408027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.206271 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f483232bfc8f4b918c4442609ec1a05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.206326 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f483232bfc8f4b918c4442609ec1a05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.206326 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f483232bfc8f4b918c4442609ec1a05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.218930 2645 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Bootstrap starting.
I20260619 00:52:50.221509 2646 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Bootstrap starting.
I20260619 00:52:50.222731 2645 tablet_bootstrap.cc:654] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.223312 2647 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Bootstrap starting.
I20260619 00:52:50.224203 2645 log.cc:824] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:50.225178 2646 tablet_bootstrap.cc:654] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.226408 2646 log.cc:824] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:50.226713 2647 tablet_bootstrap.cc:654] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.227736 2647 log.cc:824] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:50.227945 2645 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: No bootstrap required, opened a new log
I20260619 00:52:50.228299 2645 ts_tablet_manager.cc:1397] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 00:52:50.230683 2646 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: No bootstrap required, opened a new log
I20260619 00:52:50.230954 2647 tablet_bootstrap.cc:492] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: No bootstrap required, opened a new log
I20260619 00:52:50.230997 2646 ts_tablet_manager.cc:1397] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 00:52:50.231230 2647 ts_tablet_manager.cc:1397] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 00:52:50.239041 2646 raft_consensus.cc:348] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.239053 2645 raft_consensus.cc:348] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.239430 2646 raft_consensus.cc:374] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.239457 2645 raft_consensus.cc:374] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.239523 2646 raft_consensus.cc:729] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.239553 2645 raft_consensus.cc:729] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.240135 2646 consensus_queue.cc:260] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.240224 2645 consensus_queue.cc:260] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.241997 2646 ts_tablet_manager.cc:1428] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 00:52:50.241997 2647 raft_consensus.cc:348] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.242552 2647 raft_consensus.cc:374] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.242693 2647 raft_consensus.cc:729] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e303399dba40459a33885dc8871f14, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.243549 2647 consensus_queue.cc:260] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.244743 2487 heartbeater.cc:503] Master 127.0.97.62:41367 was elected leader, sending a full tablet report...
I20260619 00:52:50.245863 2647 ts_tablet_manager.cc:1428] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260619 00:52:50.247411 2626 heartbeater.cc:503] Master 127.0.97.62:41367 was elected leader, sending a full tablet report...
I20260619 00:52:50.248400 2645 ts_tablet_manager.cc:1428] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
W20260619 00:52:50.276481 2349 tablet.cc:2369] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:52:50.334548 2488 tablet.cc:2369] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:50.337828 2653 raft_consensus.cc:482] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:50.338173 2653 raft_consensus.cc:504] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.341269 2653 leader_election.cc:288] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:50.347059 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.340559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38635 (local address 127.0.97.1:38719)
0619 00:52:50.340776 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.340783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.341338 (+ 555us) server_negotiation.cc:407] Connection header received
0619 00:52:50.341662 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.341669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.341746 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.341843 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.342380 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.343162 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.345235 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.346062 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.346229 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.346343 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.346689 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.346813 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 00:52:50.347282 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.340305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.97.2:38635)
0619 00:52:50.341209 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.341231 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.341440 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.341877 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.341882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.341910 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.342239 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.342247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.343318 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.343324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.345091 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.345102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.346303 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.346309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.346535 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.346560 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.346663 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.346794 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":54,"thread_start_us":495,"threads_started":1}
I20260619 00:52:50.347811 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.340977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40219 (local address 127.0.97.3:41717)
0619 00:52:50.341205 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.341211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.341923 (+ 712us) server_negotiation.cc:407] Connection header received
0619 00:52:50.342139 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.342145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.342211 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.342309 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.342855 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.343697 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.345694 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.346733 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.346875 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.346992 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.347511 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.347643 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 00:52:50.348387 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.340478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41717 (local address 127.0.97.2:40219)
0619 00:52:50.341803 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.341819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.341990 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.342321 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.342326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.342347 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.342689 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.342697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.343833 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.343846 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.345549 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.345560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.346991 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.346999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.347286 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.347321 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.348013 (+ 692us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.348125 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":18,"thread_start_us":1012,"threads_started":1}
I20260619 00:52:50.348263 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f483232bfc8f4b918c4442609ec1a05a" candidate_uuid: "b2e303399dba40459a33885dc8871f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913" is_pre_election: true
I20260619 00:52:50.348855 2298 raft_consensus.cc:2454] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e303399dba40459a33885dc8871f14 in term 0.
I20260619 00:52:50.349392 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f483232bfc8f4b918c4442609ec1a05a" candidate_uuid: "b2e303399dba40459a33885dc8871f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b" is_pre_election: true
I20260619 00:52:50.349571 2368 leader_election.cc:302] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e303399dba40459a33885dc8871f14, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.349911 2581 raft_consensus.cc:2454] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e303399dba40459a33885dc8871f14 in term 0.
I20260619 00:52:50.349996 2653 raft_consensus.cc:2781] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:50.350131 2653 raft_consensus.cc:482] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:50.350236 2653 raft_consensus.cc:3037] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.352797 2653 raft_consensus.cc:504] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.353432 2653 leader_election.cc:288] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [CANDIDATE]: Term 1 election: Requested vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:50.353940 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f483232bfc8f4b918c4442609ec1a05a" candidate_uuid: "b2e303399dba40459a33885dc8871f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913"
I20260619 00:52:50.353966 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f483232bfc8f4b918c4442609ec1a05a" candidate_uuid: "b2e303399dba40459a33885dc8871f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b"
I20260619 00:52:50.354161 2581 raft_consensus.cc:3037] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.354161 2298 raft_consensus.cc:3037] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Advancing to term 1
W20260619 00:52:50.355176 2627 tablet.cc:2369] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:52:50.357417 2581 raft_consensus.cc:2454] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e303399dba40459a33885dc8871f14 in term 1.
I20260619 00:52:50.357417 2298 raft_consensus.cc:2454] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e303399dba40459a33885dc8871f14 in term 1.
I20260619 00:52:50.357820 2368 leader_election.cc:302] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e303399dba40459a33885dc8871f14, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.358137 2653 raft_consensus.cc:2781] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:50.358579 2653 raft_consensus.cc:686] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 1 LEADER]: Becoming Leader. State: Replica: b2e303399dba40459a33885dc8871f14, State: Running, Role: LEADER
I20260619 00:52:50.359040 2653 consensus_queue.cc:237] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.363451 2141 catalog_manager.cc:5495] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e303399dba40459a33885dc8871f14 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b2e303399dba40459a33885dc8871f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:52:50 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:50.423094 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.407411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.0.97.62:41367)
0619 00:52:50.407616 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.407624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.408298 (+ 674us) server_negotiation.cc:407] Connection header received
0619 00:52:50.408462 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.408466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.408526 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.408612 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:50.410196 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.410989 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.412908 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.413402 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.413521 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.417561 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:50.417566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:50.417569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:50.417598 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:50.420328 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.421138 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.421150 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.421152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.421266 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:50.421639 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:50.421644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:50.421647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:50.421898 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:50.422045 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.422456 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.422783 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":67}
I20260619 00:52:50.423115 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.407401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41367 (local address 127.0.0.1:34166)
0619 00:52:50.408184 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.408204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.408349 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.408679 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.408684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.409671 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:50.410024 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.410036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.411112 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.411118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.412728 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.412746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.413633 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.413640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.413831 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.414540 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:50.414562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:50.417380 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:50.420470 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.420482 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.420491 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.421019 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.421370 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:50.421376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:50.421379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:50.421532 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:50.422032 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:50.422040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:50.422196 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.422351 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.422658 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":381,"threads_started":1}
I20260619 00:52:50.428117 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34166:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 00:52:50.429252 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:50.435742 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50527241d9c44c40a88f006c509b6ac3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:50.435704 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50527241d9c44c40a88f006c509b6ac3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:50.435901 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50527241d9c44c40a88f006c509b6ac3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:52:50.436583 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.436638 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.436640 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01bfecf2c6eb474ba3bcc65c8c8bef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.441810 2647 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: Bootstrap starting.
I20260619 00:52:50.443459 2647 tablet_bootstrap.cc:654] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.443918 2646 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: Bootstrap starting.
I20260619 00:52:50.444566 2645 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: Bootstrap starting.
I20260619 00:52:50.446081 2645 tablet_bootstrap.cc:654] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.446125 2646 tablet_bootstrap.cc:654] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.448397 2645 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: No bootstrap required, opened a new log
I20260619 00:52:50.448555 2645 ts_tablet_manager.cc:1397] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 00:52:50.448781 2647 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: No bootstrap required, opened a new log
I20260619 00:52:50.448967 2647 ts_tablet_manager.cc:1397] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260619 00:52:50.449097 2646 tablet_bootstrap.cc:492] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: No bootstrap required, opened a new log
I20260619 00:52:50.449237 2646 ts_tablet_manager.cc:1397] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 00:52:50.449362 2645 raft_consensus.cc:348] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.449633 2645 raft_consensus.cc:374] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.449730 2645 raft_consensus.cc:729] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.449819 2647 raft_consensus.cc:348] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.450042 2647 raft_consensus.cc:374] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.449956 2645 consensus_queue.cc:260] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.450119 2647 raft_consensus.cc:729] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e303399dba40459a33885dc8871f14, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.450035 2646 raft_consensus.cc:348] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.450255 2646 raft_consensus.cc:374] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.450350 2646 raft_consensus.cc:729] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.450317 2647 consensus_queue.cc:260] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.450573 2645 ts_tablet_manager.cc:1428] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:50.450645 2646 consensus_queue.cc:260] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.450917 2647 ts_tablet_manager.cc:1428] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:50.451259 2646 ts_tablet_manager.cc:1428] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:50.451305 2652 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.451679 2652 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.452013 2653 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.452263 2653 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.453081 2651 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.453438 2651 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.494837 2651 raft_consensus.cc:482] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:50.495070 2651 raft_consensus.cc:504] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.496706 2651 leader_election.cc:288] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:50.503669 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.497178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56821 (local address 127.0.97.3:41717)
0619 00:52:50.497372 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.497380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.498375 (+ 995us) server_negotiation.cc:407] Connection header received
0619 00:52:50.498529 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.498532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.498587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.498655 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.499229 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.499985 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.501909 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.502694 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.502852 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.502962 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.503246 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.503381 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:52:50.503682 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.497134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57473 (local address 127.0.97.2:45887)
0619 00:52:50.497363 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.497373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.498116 (+ 743us) server_negotiation.cc:407] Connection header received
0619 00:52:50.498305 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.498314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.498401 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.498493 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.499341 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.500067 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.501885 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.502656 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.502811 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.502940 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.503234 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.503371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:52:50.503988 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.497138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41717 (local address 127.0.97.1:56821)
0619 00:52:50.498268 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.498285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.498405 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.498703 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.498706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.498725 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.499052 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.499061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.500121 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.500129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.501747 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.501758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.502934 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.502940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.503093 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.503115 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.503219 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.503372 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":31,"thread_start_us":779,"threads_started":1}
I20260619 00:52:50.504026 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.497073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45887 (local address 127.0.97.1:57473)
0619 00:52:50.497985 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.498007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.498156 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.498774 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.498777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.498808 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.499168 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.499175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.500205 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.500212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.501728 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.501743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.502900 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.502913 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.503078 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.503102 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.503209 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.503395 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":66,"thread_start_us":562,"threads_started":1}
I20260619 00:52:50.504714 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bfecf2c6eb474ba3bcc65c8c8bef81" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b" is_pre_election: true
I20260619 00:52:50.505035 2581 raft_consensus.cc:2454] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 0.
I20260619 00:52:50.505103 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bfecf2c6eb474ba3bcc65c8c8bef81" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14" is_pre_election: true
I20260619 00:52:50.505527 2442 raft_consensus.cc:2454] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 0.
I20260619 00:52:50.505540 2231 leader_election.cc:302] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.505945 2651 raft_consensus.cc:2781] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:50.506091 2651 raft_consensus.cc:482] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:50.506192 2651 raft_consensus.cc:3037] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.508278 2651 raft_consensus.cc:504] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.508889 2651 leader_election.cc:288] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 election: Requested vote from peers b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:50.509326 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bfecf2c6eb474ba3bcc65c8c8bef81" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14"
I20260619 00:52:50.509332 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01bfecf2c6eb474ba3bcc65c8c8bef81" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b"
I20260619 00:52:50.509516 2442 raft_consensus.cc:3037] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.509552 2581 raft_consensus.cc:3037] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.511806 2581 raft_consensus.cc:2454] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 1.
I20260619 00:52:50.511981 2442 raft_consensus.cc:2454] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 1.
I20260619 00:52:50.512226 2231 leader_election.cc:302] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.512497 2651 raft_consensus.cc:2781] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:50.513029 2651 raft_consensus.cc:686] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 LEADER]: Becoming Leader. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Running, Role: LEADER
I20260619 00:52:50.513551 2651 consensus_queue.cc:237] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.516923 2669 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: TxnStatusTablet state changed. Reason: New leader ccc9ae1782e2477e9a4f8860514c8913. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.517405 2669 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:50.518066 2671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 00:52:50.518325 2671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 00:52:50.518586 2140 catalog_manager.cc:5495] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } health_report { overall_health: UNKNOWN } } }
I20260619 00:52:50.540972 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.534491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.0.1:55402)
0619 00:52:50.534741 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.534761 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.534883 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.535329 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.535333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.535343 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:50.535512 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.535520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.537011 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.537015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.537983 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.538021 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.539037 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.539042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.539208 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.539271 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.540405 (+ 1134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.540435 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.540523 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.540650 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 00:52:50.541025 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.534695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.0.97.1:38719)
0619 00:52:50.534886 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.534892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.534913 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:52:50.535053 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.535059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.535158 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.535317 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.535644 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.536870 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.538164 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.538843 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.538977 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.539490 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.540266 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.540417 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.540566 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.540660 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:52:50.550016 2581 raft_consensus.cc:1261] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Refusing update from remote peer ccc9ae1782e2477e9a4f8860514c8913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.550072 2442 raft_consensus.cc:1261] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Refusing update from remote peer ccc9ae1782e2477e9a4f8860514c8913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.551115 2669 consensus_queue.cc:1035] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.551555 2651 consensus_queue.cc:1035] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.560395 2652 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: TxnStatusTablet state changed. Reason: New leader ccc9ae1782e2477e9a4f8860514c8913. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.560655 2652 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.563572 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298377197763031040 to 7298377197627412480. Current Snapshot: MvccSnapshot[applied={T|T < 7298377197763031040}]
I20260619 00:52:50.565672 2652 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.565897 2652 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P 38cd251d4d404145bea2cab1232e054b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.571802 2653 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: TxnStatusTablet state changed. Reason: New leader ccc9ae1782e2477e9a4f8860514c8913. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.572149 2653 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.573101 2669 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.573378 2669 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:50.577569 2653 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.577858 2653 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P b2e303399dba40459a33885dc8871f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:52:50.583866 2651 tablet_replica.cc:439] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } } }
I20260619 00:52:50.584177 2651 tablet_replica.cc:442] T 01bfecf2c6eb474ba3bcc65c8c8bef81 P ccc9ae1782e2477e9a4f8860514c8913: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:52:50.601416 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.594219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.97.2:45887)
0619 00:52:50.594400 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.594406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.594813 (+ 407us) server_negotiation.cc:407] Connection header received
0619 00:52:50.595046 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.595051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.595165 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.595317 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.595693 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.596852 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.598812 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.599448 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.599567 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.600048 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.600748 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.600910 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.601036 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.601113 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:52:50.622970 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.613015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.97.2:57419)
0619 00:52:50.617349 (+ 4334us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.617372 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.617504 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.617870 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.617874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.617893 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:50.618165 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.618172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.619636 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.619640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.620759 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.620771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.621848 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.621853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.621962 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.622017 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.622472 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.622483 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.622573 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.622674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4182,"thread_start_us":4017,"threads_started":1}
I20260619 00:52:50.622974 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.613115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57419 (local address 127.0.97.1:38719)
0619 00:52:50.613332 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.613339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.617460 (+ 4121us) server_negotiation.cc:407] Connection header received
0619 00:52:50.617624 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.617630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.617736 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.617842 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.618308 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.619473 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.620979 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.621652 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.621779 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.622203 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.622346 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.622443 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.622613 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.622693 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 00:52:50.633059 2581 raft_consensus.cc:1261] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Refusing update from remote peer b2e303399dba40459a33885dc8871f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.633368 2298 raft_consensus.cc:1261] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Refusing update from remote peer b2e303399dba40459a33885dc8871f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.633736 2656 consensus_queue.cc:1035] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.634101 2653 consensus_queue.cc:1035] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.684574 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.677698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59813 (local address 127.0.97.2:45887)
0619 00:52:50.677924 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.677932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.678493 (+ 561us) server_negotiation.cc:407] Connection header received
0619 00:52:50.678693 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.678703 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.678775 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.678847 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.679468 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.680434 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.682714 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.683263 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.683403 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.684011 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.684107 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.684177 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.684313 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.684372 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:52:50.684859 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.677555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45887 (local address 127.0.97.1:59813)
0619 00:52:50.678376 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.678401 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.678552 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.678887 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.678892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.678904 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:50.679298 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.679315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.680566 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.680572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.682559 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.682575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.683539 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.683546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.683728 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.683823 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.684205 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.684216 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.684309 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.684458 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":451,"threads_started":1}
I20260619 00:52:50.724581 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:50.724920 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:50.728758 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Sending DeleteTablet for 3 replicas of tablet f483232bfc8f4b918c4442609ec1a05a
I20260619 00:52:50.729925 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet f483232bfc8f4b918c4442609ec1a05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260619 00:52:50.730571 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet f483232bfc8f4b918c4442609ec1a05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35400
I20260619 00:52:50.732080 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:50.732709 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:50.734428 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet f483232bfc8f4b918c4442609ec1a05a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260619 00:52:50.741899 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 (DEFAULT_TABLE table=test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.742007 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 (DEFAULT_TABLE table=test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.742425 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ef8b7dfa324e5486eaa0b2006bd322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.742437 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ef8b7dfa324e5486eaa0b2006bd322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.745625 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 (DEFAULT_TABLE table=test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:50.746188 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ef8b7dfa324e5486eaa0b2006bd322. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:50.753553 2645 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Bootstrap starting.
I20260619 00:52:50.755266 2645 tablet_bootstrap.cc:654] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.757354 2646 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Bootstrap starting.
I20260619 00:52:50.759147 2646 tablet_bootstrap.cc:654] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.765416 2647 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Bootstrap starting.
I20260619 00:52:50.767349 2647 tablet_bootstrap.cc:654] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:50.791754 2646 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: No bootstrap required, opened a new log
I20260619 00:52:50.792083 2646 ts_tablet_manager.cc:1397] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20260619 00:52:50.793025 2646 raft_consensus.cc:348] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.793437 2646 raft_consensus.cc:374] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.793572 2646 raft_consensus.cc:729] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.793944 2646 consensus_queue.cc:260] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.794874 2646 ts_tablet_manager.cc:1428] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 00:52:50.796272 2647 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: No bootstrap required, opened a new log
I20260619 00:52:50.796512 2647 ts_tablet_manager.cc:1397] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.004s
I20260619 00:52:50.797407 2647 raft_consensus.cc:348] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.797729 2647 raft_consensus.cc:374] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.797854 2647 raft_consensus.cc:729] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e303399dba40459a33885dc8871f14, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.798223 2647 consensus_queue.cc:260] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.798974 2647 ts_tablet_manager.cc:1428] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:50.801545 2697 tablet_replica.cc:330] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: stopping tablet replica
I20260619 00:52:50.802057 2697 raft_consensus.cc:2229] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:50.802763 2697 raft_consensus.cc:2258] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:50.804780 2697 ts_tablet_manager.cc:1905] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:50.805394 2698 tablet_replica.cc:330] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: stopping tablet replica
I20260619 00:52:50.806740 2698 raft_consensus.cc:2229] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:50.806820 2645 tablet_bootstrap.cc:492] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: No bootstrap required, opened a new log
I20260619 00:52:50.807004 2645 ts_tablet_manager.cc:1397] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.009s
I20260619 00:52:50.807204 2698 raft_consensus.cc:2258] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:50.807744 2645 raft_consensus.cc:348] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.808022 2645 raft_consensus.cc:374] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:50.808087 2645 raft_consensus.cc:729] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Initialized, Role: FOLLOWER
I20260619 00:52:50.808866 2696 tablet_replica.cc:330] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: stopping tablet replica
I20260619 00:52:50.809283 2696 raft_consensus.cc:2229] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:50.810209 2696 raft_consensus.cc:2258] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:50.809677 2645 consensus_queue.cc:260] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.810825 2645 ts_tablet_manager.cc:1428] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260619 00:52:50.812392 2698 ts_tablet_manager.cc:1905] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:50.813745 2696 ts_tablet_manager.cc:1905] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:50.816619 2652 raft_consensus.cc:482] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:50.816802 2652 raft_consensus.cc:504] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.818591 2652 leader_election.cc:288] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887)
I20260619 00:52:50.821558 2696 ts_tablet_manager.cc:1918] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:50.821772 2696 log.cc:1196] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/f483232bfc8f4b918c4442609ec1a05a
I20260619 00:52:50.822397 2696 ts_tablet_manager.cc:1939] T f483232bfc8f4b918c4442609ec1a05a P 38cd251d4d404145bea2cab1232e054b: Deleting consensus metadata
I20260619 00:52:50.822535 2697 ts_tablet_manager.cc:1918] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:50.822778 2697 log.cc:1196] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/f483232bfc8f4b918c4442609ec1a05a
I20260619 00:52:50.823335 2697 ts_tablet_manager.cc:1939] T f483232bfc8f4b918c4442609ec1a05a P b2e303399dba40459a33885dc8871f14: Deleting consensus metadata
I20260619 00:52:50.824667 2128 catalog_manager.cc:4841] TS 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717): tablet f483232bfc8f4b918c4442609ec1a05a (table test-table [id=ab95d2c8340b4c90a32d12e812408027]) successfully deleted
I20260619 00:52:50.825114 2125 catalog_manager.cc:4841] TS b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887): tablet f483232bfc8f4b918c4442609ec1a05a (table test-table [id=ab95d2c8340b4c90a32d12e812408027]) successfully deleted
I20260619 00:52:50.833710 2698 ts_tablet_manager.cc:1918] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:50.833896 2698 log.cc:1196] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/f483232bfc8f4b918c4442609ec1a05a
I20260619 00:52:50.834427 2698 ts_tablet_manager.cc:1939] T f483232bfc8f4b918c4442609ec1a05a P ccc9ae1782e2477e9a4f8860514c8913: Deleting consensus metadata
I20260619 00:52:50.836160 2125 catalog_manager.cc:4841] TS ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719): tablet f483232bfc8f4b918c4442609ec1a05a (table test-table [id=ab95d2c8340b4c90a32d12e812408027]) successfully deleted
I20260619 00:52:50.846383 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.819396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43041 (local address 127.0.97.1:38719)
0619 00:52:50.819593 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.819599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.837183 (+ 17584us) server_negotiation.cc:407] Connection header received
0619 00:52:50.837371 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.837377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.837475 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.837601 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.838134 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.839268 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.841882 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.845037 (+ 3155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.845145 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.845230 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.845893 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.846039 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:52:50.847250 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.819073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.97.3:43041)
0619 00:52:50.837080 (+ 18007us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.837102 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.837254 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.837621 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.837626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.837656 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.837958 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.837971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.839469 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.839475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.841692 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.841708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.845428 (+ 3720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.845433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.845627 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.845688 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.846655 (+ 967us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.846831 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17868,"thread_start_us":17748,"threads_started":1}
I20260619 00:52:50.847958 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ef8b7dfa324e5486eaa0b2006bd322" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913" is_pre_election: true
I20260619 00:52:50.848265 2298 raft_consensus.cc:2454] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 0.
I20260619 00:52:50.849143 2507 leader_election.cc:302] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.849596 2652 raft_consensus.cc:2781] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:50.849732 2652 raft_consensus.cc:482] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:50.849838 2652 raft_consensus.cc:3037] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.852465 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.837343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35085 (local address 127.0.97.2:45887)
0619 00:52:50.837517 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.837525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.844309 (+ 6784us) server_negotiation.cc:407] Connection header received
0619 00:52:50.844723 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.844733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.844829 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.844905 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:52:50.845751 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.846891 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.849599 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.851006 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.851126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.851237 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.852012 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.852172 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:52:50.853425 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.837656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45887 (local address 127.0.97.3:35085)
0619 00:52:50.844203 (+ 6547us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.844223 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.844558 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.845109 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.845113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.845141 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:52:50.845543 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.845556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.847166 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.847173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.849400 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.849415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.851499 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.851511 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.851735 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.851774 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.852737 (+ 963us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.852911 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6426,"thread_start_us":6655,"threads_started":1}
I20260619 00:52:50.853449 2652 raft_consensus.cc:504] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.854187 2652 leader_election.cc:288] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 election: Requested vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887)
I20260619 00:52:50.855086 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ef8b7dfa324e5486eaa0b2006bd322" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913"
I20260619 00:52:50.855083 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ef8b7dfa324e5486eaa0b2006bd322" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14" is_pre_election: true
I20260619 00:52:50.855341 2298 raft_consensus.cc:3037] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.855365 2442 raft_consensus.cc:2454] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 0.
I20260619 00:52:50.857910 2298 raft_consensus.cc:2454] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 1.
I20260619 00:52:50.858492 2507 leader_election.cc:302] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:50.858739 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ef8b7dfa324e5486eaa0b2006bd322" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14"
I20260619 00:52:50.858968 2442 raft_consensus.cc:3037] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:50.859198 2652 raft_consensus.cc:2781] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:50.865614 2442 raft_consensus.cc:2454] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 1.
I20260619 00:52:50.867264 2652 raft_consensus.cc:686] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 1 LEADER]: Becoming Leader. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Running, Role: LEADER
I20260619 00:52:50.867707 2652 consensus_queue.cc:237] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:50.871790 2139 catalog_manager.cc:5495] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b reported cstate change: term changed from 0 to 1, leader changed from <none> to 38cd251d4d404145bea2cab1232e054b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "38cd251d4d404145bea2cab1232e054b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } health_report { overall_health: HEALTHY } } }
I20260619 00:52:50.908174 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.902419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.97.3:41717)
0619 00:52:50.902644 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.902650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.902832 (+ 182us) server_negotiation.cc:407] Connection header received
0619 00:52:50.903108 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.903111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.903189 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.903338 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.903710 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.904420 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.905773 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.906187 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.906503 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.906990 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.907556 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.907713 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.907802 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.907880 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:52:50.924101 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.918263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36337 (local address 127.0.97.1:38719)
0619 00:52:50.918510 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.918520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.919121 (+ 601us) server_negotiation.cc:407] Connection header received
0619 00:52:50.919330 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.919335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.919440 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.919539 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.920021 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.920856 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.922392 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.922852 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.922984 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.923431 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.923536 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.923635 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.923790 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.923882 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:52:50.924760 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.918172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.97.3:36337)
0619 00:52:50.919019 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.919035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.919197 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.919551 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.919555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.919567 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:50.919874 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.919882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.920989 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.920996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.922234 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.922245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.923036 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.923044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.923178 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.923229 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.923647 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.923659 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.924389 (+ 730us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.924505 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":530,"threads_started":1}
I20260619 00:52:50.935422 2442 raft_consensus.cc:1261] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Refusing update from remote peer 38cd251d4d404145bea2cab1232e054b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.935967 2298 raft_consensus.cc:1261] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Refusing update from remote peer 38cd251d4d404145bea2cab1232e054b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:50.936126 2704 consensus_queue.cc:1035] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.936833 2704 consensus_queue.cc:1035] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:50.956003 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298377199345442816 to 7298377199078895616. Current Snapshot: MvccSnapshot[applied={T|T < 7298377199345442816}]
I20260619 00:52:50.996608 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.989310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41717 (local address 127.0.97.1:47961)
0619 00:52:50.989597 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:50.989622 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:50.989736 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:50.990326 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:50.990331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:50.990344 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:50.990542 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.990552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.992055 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.992062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.993936 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:50.993954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.995161 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.995173 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:50.995383 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.995437 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.995976 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.995988 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 00:52:50.996077 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 00:52:50.996241 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 00:52:50.997347 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:50.989733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47961 (local address 127.0.97.3:41717)
0619 00:52:50.989895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:50.989901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:50.989920 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:52:50.989987 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:50.989993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:50.990076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:50.990171 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:50.990715 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.991864 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.994138 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:50.994885 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:50.995025 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:50.995662 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:50.995829 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:50.995922 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:50.996949 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0619 00:52:50.997034 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 00:52:50.998422 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 00:52:52.637602 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:52.637856 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:52.641705 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Sending DeleteTablet for 3 replicas of tablet 33ef8b7dfa324e5486eaa0b2006bd322
I20260619 00:52:52.642581 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260619 00:52:52.642675 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35400
I20260619 00:52:52.642730 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ef8b7dfa324e5486eaa0b2006bd322 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260619 00:52:52.655570 2738 tablet_replica.cc:330] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: stopping tablet replica
I20260619 00:52:52.655632 2739 tablet_replica.cc:330] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: stopping tablet replica
I20260619 00:52:52.655797 2740 tablet_replica.cc:330] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: stopping tablet replica
I20260619 00:52:52.655820 2738 raft_consensus.cc:2229] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:52.655939 2739 raft_consensus.cc:2229] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:52.656071 2740 raft_consensus.cc:2229] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:52.656203 2738 raft_consensus.cc:2258] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.656286 2739 raft_consensus.cc:2258] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.656738 2740 raft_consensus.cc:2258] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.657262 2738 ts_tablet_manager.cc:1905] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.657611 2739 ts_tablet_manager.cc:1905] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.657792 2740 ts_tablet_manager.cc:1905] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.664217 2738 ts_tablet_manager.cc:1918] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:52.664464 2738 log.cc:1196] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/33ef8b7dfa324e5486eaa0b2006bd322
I20260619 00:52:52.664595 2740 ts_tablet_manager.cc:1918] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:52.664788 2740 log.cc:1196] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/33ef8b7dfa324e5486eaa0b2006bd322
I20260619 00:52:52.665132 2738 ts_tablet_manager.cc:1939] T 33ef8b7dfa324e5486eaa0b2006bd322 P ccc9ae1782e2477e9a4f8860514c8913: Deleting consensus metadata
I20260619 00:52:52.665307 2740 ts_tablet_manager.cc:1939] T 33ef8b7dfa324e5486eaa0b2006bd322 P 38cd251d4d404145bea2cab1232e054b: Deleting consensus metadata
I20260619 00:52:52.665467 2739 ts_tablet_manager.cc:1918] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:52.665688 2739 log.cc:1196] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/33ef8b7dfa324e5486eaa0b2006bd322
I20260619 00:52:52.666234 2739 ts_tablet_manager.cc:1939] T 33ef8b7dfa324e5486eaa0b2006bd322 P b2e303399dba40459a33885dc8871f14: Deleting consensus metadata
I20260619 00:52:52.666376 2128 catalog_manager.cc:4841] TS 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717): tablet 33ef8b7dfa324e5486eaa0b2006bd322 (table test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]) successfully deleted
I20260619 00:52:52.666376 2125 catalog_manager.cc:4841] TS ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719): tablet 33ef8b7dfa324e5486eaa0b2006bd322 (table test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]) successfully deleted
I20260619 00:52:52.667483 2125 catalog_manager.cc:4841] TS b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887): tablet 33ef8b7dfa324e5486eaa0b2006bd322 (table test-table [id=a8ed631eeb2d4cd8afcfab310292efd2]) successfully deleted
Jun 19 00:52:52 dist-test-slave-r7bh krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:52:52 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:52.708020 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:52.689453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.97.62:41367)
0619 00:52:52.689998 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:52.690003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:52.690026 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:52:52.690129 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:52.690132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:52.690192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:52.690287 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:52.691722 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:52.692591 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:52.694026 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:52.694443 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:52.694553 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:52.702389 (+ 7836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:52.702396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:52.702400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:52.702444 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:52.705658 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:52.706338 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:52.706342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:52.706344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:52.706501 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:52.706830 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:52.706833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:52.706835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:52.707113 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:52.707285 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:52.707534 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 00:52:52.707700 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260619 00:52:52.711925 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:52.712638 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:52.719444 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 2362fcea71594027a8984c65dc2976bc (DEFAULT_TABLE table=test-table [id=fe725075a4504163bdcbae52e0230900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.719550 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 2362fcea71594027a8984c65dc2976bc (DEFAULT_TABLE table=test-table [id=fe725075a4504163bdcbae52e0230900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.719554 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 2362fcea71594027a8984c65dc2976bc (DEFAULT_TABLE table=test-table [id=fe725075a4504163bdcbae52e0230900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.719933 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362fcea71594027a8984c65dc2976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.719933 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362fcea71594027a8984c65dc2976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.719981 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2362fcea71594027a8984c65dc2976bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.724689 2749 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Bootstrap starting.
I20260619 00:52:52.726435 2749 tablet_bootstrap.cc:654] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.727496 2751 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Bootstrap starting.
I20260619 00:52:52.729838 2751 tablet_bootstrap.cc:654] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.730546 2750 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Bootstrap starting.
I20260619 00:52:52.731667 2749 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: No bootstrap required, opened a new log
I20260619 00:52:52.731902 2749 ts_tablet_manager.cc:1397] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260619 00:52:52.732301 2750 tablet_bootstrap.cc:654] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.732944 2751 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: No bootstrap required, opened a new log
I20260619 00:52:52.733186 2751 ts_tablet_manager.cc:1397] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 00:52:52.733973 2751 raft_consensus.cc:348] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.733603 2749 raft_consensus.cc:348] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.734264 2751 raft_consensus.cc:374] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.734294 2749 raft_consensus.cc:374] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.734376 2751 raft_consensus.cc:729] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.734498 2749 raft_consensus.cc:729] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e303399dba40459a33885dc8871f14, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.734664 2751 consensus_queue.cc:260] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.734756 2749 consensus_queue.cc:260] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.735014 2750 tablet_bootstrap.cc:492] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: No bootstrap required, opened a new log
I20260619 00:52:52.735185 2750 ts_tablet_manager.cc:1397] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 00:52:52.735410 2751 ts_tablet_manager.cc:1428] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:52.736020 2750 raft_consensus.cc:348] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.736686 2749 ts_tablet_manager.cc:1428] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 00:52:52.736950 2750 raft_consensus.cc:374] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.737115 2750 raft_consensus.cc:729] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.737425 2750 consensus_queue.cc:260] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.738108 2750 ts_tablet_manager.cc:1428] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 00:52:52.834144 2729 raft_consensus.cc:482] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:52.834350 2729 raft_consensus.cc:504] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.835065 2729 leader_election.cc:288] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887)
I20260619 00:52:52.835641 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362fcea71594027a8984c65dc2976bc" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913" is_pre_election: true
I20260619 00:52:52.835714 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362fcea71594027a8984c65dc2976bc" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14" is_pre_election: true
I20260619 00:52:52.835916 2297 raft_consensus.cc:2454] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 0.
I20260619 00:52:52.835966 2442 raft_consensus.cc:2454] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 0.
I20260619 00:52:52.836467 2507 leader_election.cc:302] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:52.836757 2729 raft_consensus.cc:2781] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:52.836910 2729 raft_consensus.cc:482] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:52.837015 2729 raft_consensus.cc:3037] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.839247 2729 raft_consensus.cc:504] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.839795 2729 leader_election.cc:288] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 election: Requested vote from peers ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719), b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887)
I20260619 00:52:52.840317 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362fcea71594027a8984c65dc2976bc" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc9ae1782e2477e9a4f8860514c8913"
I20260619 00:52:52.840580 2297 raft_consensus.cc:3037] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.840577 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2362fcea71594027a8984c65dc2976bc" candidate_uuid: "38cd251d4d404145bea2cab1232e054b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14"
I20260619 00:52:52.840873 2442 raft_consensus.cc:3037] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.844141 2297 raft_consensus.cc:2454] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 1.
I20260619 00:52:52.844142 2442 raft_consensus.cc:2454] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38cd251d4d404145bea2cab1232e054b in term 1.
I20260619 00:52:52.844568 2507 leader_election.cc:302] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:52.844866 2729 raft_consensus.cc:2781] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:52.845011 2729 raft_consensus.cc:686] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 1 LEADER]: Becoming Leader. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Running, Role: LEADER
I20260619 00:52:52.845314 2729 consensus_queue.cc:237] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.848044 2140 catalog_manager.cc:5495] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b reported cstate change: term changed from 0 to 1, leader changed from <none> to 38cd251d4d404145bea2cab1232e054b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "38cd251d4d404145bea2cab1232e054b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } health_report { overall_health: HEALTHY } } }
I20260619 00:52:52.871186 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:52.866122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.0.97.3:41717)
0619 00:52:52.866728 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:52.866739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:52.866756 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:52:52.866807 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:52.866810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:52.866872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:52.866986 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:52.867382 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:52.868147 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:52.869475 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:52.869880 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:52.870017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:52.870434 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:52.870546 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:52.870647 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:52.870794 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 00:52:52.870866 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
I20260619 00:52:52.879984 2297 raft_consensus.cc:1261] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Refusing update from remote peer 38cd251d4d404145bea2cab1232e054b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:52.880038 2442 raft_consensus.cc:1261] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Refusing update from remote peer 38cd251d4d404145bea2cab1232e054b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:52.880528 2711 consensus_queue.cc:1035] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:52.880853 2729 consensus_queue.cc:1035] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:52.885527 2721 mvcc.cc:205] Tried to move back new op lower bound from 7298377207312134144 to 7298377207176708096. Current Snapshot: MvccSnapshot[applied={T|T < 7298377207312134144}]
I20260619 00:52:52.932179 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:52.932492 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:52.936102 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Sending DeleteTablet for 3 replicas of tablet 2362fcea71594027a8984c65dc2976bc
I20260619 00:52:52.937377 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362fcea71594027a8984c65dc2976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35400
I20260619 00:52:52.937487 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362fcea71594027a8984c65dc2976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260619 00:52:52.937582 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 2362fcea71594027a8984c65dc2976bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260619 00:52:52.937670 2739 tablet_replica.cc:330] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: stopping tablet replica
I20260619 00:52:52.937768 2738 tablet_replica.cc:330] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: stopping tablet replica
I20260619 00:52:52.937846 2740 tablet_replica.cc:330] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: stopping tablet replica
I20260619 00:52:52.937877 2739 raft_consensus.cc:2229] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:52.937971 2738 raft_consensus.cc:2229] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:52.938040 2740 raft_consensus.cc:2229] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:52.938082 2739 raft_consensus.cc:2258] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.938158 2738 raft_consensus.cc:2258] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.938359 2740 raft_consensus.cc:2258] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:52.938606 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:52:52.939273 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:52:52.939296 2739 ts_tablet_manager.cc:1905] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.939352 2740 ts_tablet_manager.cc:1905] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.942386 2738 ts_tablet_manager.cc:1905] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:52.945888 2740 ts_tablet_manager.cc:1918] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:52.946084 2740 log.cc:1196] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/2362fcea71594027a8984c65dc2976bc
I20260619 00:52:52.946556 2740 ts_tablet_manager.cc:1939] T 2362fcea71594027a8984c65dc2976bc P 38cd251d4d404145bea2cab1232e054b: Deleting consensus metadata
I20260619 00:52:52.946795 2739 ts_tablet_manager.cc:1918] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:52.946945 2739 log.cc:1196] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/2362fcea71594027a8984c65dc2976bc
I20260619 00:52:52.947345 2739 ts_tablet_manager.cc:1939] T 2362fcea71594027a8984c65dc2976bc P b2e303399dba40459a33885dc8871f14: Deleting consensus metadata
I20260619 00:52:52.947844 2128 catalog_manager.cc:4841] TS 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717): tablet 2362fcea71594027a8984c65dc2976bc (table test-table [id=fe725075a4504163bdcbae52e0230900]) successfully deleted
I20260619 00:52:52.948494 2125 catalog_manager.cc:4841] TS b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887): tablet 2362fcea71594027a8984c65dc2976bc (table test-table [id=fe725075a4504163bdcbae52e0230900]) successfully deleted
I20260619 00:52:52.948938 2738 ts_tablet_manager.cc:1918] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:52:52.949079 2738 log.cc:1196] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/2362fcea71594027a8984c65dc2976bc
I20260619 00:52:52.949474 2738 ts_tablet_manager.cc:1939] T 2362fcea71594027a8984c65dc2976bc P ccc9ae1782e2477e9a4f8860514c8913: Deleting consensus metadata
I20260619 00:52:52.950448 2125 catalog_manager.cc:4841] TS ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719): tablet 2362fcea71594027a8984c65dc2976bc (table test-table [id=fe725075a4504163bdcbae52e0230900]) successfully deleted
I20260619 00:52:52.953346 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe (DEFAULT_TABLE table=test-table [id=dfbe855953ea43bd970760fad8f3961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.953464 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe (DEFAULT_TABLE table=test-table [id=dfbe855953ea43bd970760fad8f3961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.953614 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe (DEFAULT_TABLE table=test-table [id=dfbe855953ea43bd970760fad8f3961c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:52:52.953804 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b23d7977268431eaaf4397a7dfc35fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.953814 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b23d7977268431eaaf4397a7dfc35fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.953999 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b23d7977268431eaaf4397a7dfc35fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:52.958537 2749 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Bootstrap starting.
I20260619 00:52:52.960227 2750 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Bootstrap starting.
I20260619 00:52:52.960287 2749 tablet_bootstrap.cc:654] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.961644 2751 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Bootstrap starting.
I20260619 00:52:52.961773 2750 tablet_bootstrap.cc:654] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.963240 2749 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: No bootstrap required, opened a new log
I20260619 00:52:52.963393 2749 ts_tablet_manager.cc:1397] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260619 00:52:52.963433 2751 tablet_bootstrap.cc:654] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:52.963968 2749 raft_consensus.cc:348] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.964140 2749 raft_consensus.cc:374] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.964238 2749 raft_consensus.cc:729] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e303399dba40459a33885dc8871f14, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.964478 2749 consensus_queue.cc:260] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.965206 2750 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: No bootstrap required, opened a new log
I20260619 00:52:52.965870 2750 ts_tablet_manager.cc:1397] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 00:52:52.965971 2749 ts_tablet_manager.cc:1428] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 00:52:52.966589 2750 raft_consensus.cc:348] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.966990 2750 raft_consensus.cc:374] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.967083 2750 raft_consensus.cc:729] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cd251d4d404145bea2cab1232e054b, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.967355 2750 consensus_queue.cc:260] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.968397 2750 ts_tablet_manager.cc:1428] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:52.970162 2751 tablet_bootstrap.cc:492] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: No bootstrap required, opened a new log
I20260619 00:52:52.970356 2751 ts_tablet_manager.cc:1397] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 00:52:52.971097 2751 raft_consensus.cc:348] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.971359 2751 raft_consensus.cc:374] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:52.971442 2751 raft_consensus.cc:729] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Initialized, Role: FOLLOWER
I20260619 00:52:52.971702 2751 consensus_queue.cc:260] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.972682 2751 ts_tablet_manager.cc:1428] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:52:52.972905 2719 raft_consensus.cc:482] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:52:52.973070 2719 raft_consensus.cc:504] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.973785 2719 leader_election.cc:288] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:52.974212 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b23d7977268431eaaf4397a7dfc35fe" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14" is_pre_election: true
I20260619 00:52:52.974373 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b23d7977268431eaaf4397a7dfc35fe" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b" is_pre_election: true
I20260619 00:52:52.974469 2442 raft_consensus.cc:2454] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 0.
I20260619 00:52:52.974596 2581 raft_consensus.cc:2454] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 0.
I20260619 00:52:52.974920 2228 leader_election.cc:302] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e303399dba40459a33885dc8871f14, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:52.975407 2719 raft_consensus.cc:2781] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:52:52.975538 2719 raft_consensus.cc:482] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:52:52.975629 2719 raft_consensus.cc:3037] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.977707 2719 raft_consensus.cc:504] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.978344 2719 leader_election.cc:288] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 election: Requested vote from peers b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887), 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717)
I20260619 00:52:52.978932 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b23d7977268431eaaf4397a7dfc35fe" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cd251d4d404145bea2cab1232e054b"
I20260619 00:52:52.979146 2581 raft_consensus.cc:3037] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.980300 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b23d7977268431eaaf4397a7dfc35fe" candidate_uuid: "ccc9ae1782e2477e9a4f8860514c8913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e303399dba40459a33885dc8871f14"
I20260619 00:52:52.980533 2442 raft_consensus.cc:3037] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:52.982612 2581 raft_consensus.cc:2454] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 1.
I20260619 00:52:52.983130 2231 leader_election.cc:302] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cd251d4d404145bea2cab1232e054b, ccc9ae1782e2477e9a4f8860514c8913; no voters:
I20260619 00:52:52.983368 2442 raft_consensus.cc:2454] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc9ae1782e2477e9a4f8860514c8913 in term 1.
I20260619 00:52:52.983562 2719 raft_consensus.cc:2781] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:52.983745 2719 raft_consensus.cc:686] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 1 LEADER]: Becoming Leader. State: Replica: ccc9ae1782e2477e9a4f8860514c8913, State: Running, Role: LEADER
I20260619 00:52:52.984025 2719 consensus_queue.cc:237] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } }
I20260619 00:52:52.987277 2139 catalog_manager.cc:5495] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ccc9ae1782e2477e9a4f8860514c8913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccc9ae1782e2477e9a4f8860514c8913" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 } health_report { overall_health: UNKNOWN } } }
I20260619 00:52:53.012945 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:53.006106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.97.1:38719)
0619 00:52:53.006714 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:53.006722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:53.006748 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:52:53.006819 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:53.006825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:53.006919 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:53.007039 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:53.007429 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.008745 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.010203 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.010995 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.011134 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:53.011568 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:53.011726 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:53.011824 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:53.011956 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 00:52:53.012060 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":303,"threads_started":1}
I20260619 00:52:53.028664 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:53.022566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35355 (local address 127.0.97.1:38719)
0619 00:52:53.022759 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:53.022768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:53.023200 (+ 432us) server_negotiation.cc:407] Connection header received
0619 00:52:53.023345 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:53.023350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:53.023433 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:53.023527 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:52:53.023956 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.025114 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.026691 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.027347 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.027472 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:53.027859 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:52:53.027998 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:52:53.028087 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:53.028248 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 00:52:53.028352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260619 00:52:53.028687 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:53.022288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38719 (local address 127.0.97.1:35355)
0619 00:52:53.023097 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:53.023113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:53.023234 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:53.023584 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:53.023588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:53.023597 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:52:53.023816 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:53.023824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.025234 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.025240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:53.026542 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:53.026554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:53.027538 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:53.027543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:53.027665 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:53.027702 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:52:53.028120 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:52:53.028131 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:52:53.028212 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 00:52:53.028356 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":21,"thread_start_us":530,"threads_started":1}
I20260619 00:52:53.036813 2581 raft_consensus.cc:1261] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Refusing update from remote peer ccc9ae1782e2477e9a4f8860514c8913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:53.036973 2442 raft_consensus.cc:1261] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Refusing update from remote peer ccc9ae1782e2477e9a4f8860514c8913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:52:53.037428 2724 consensus_queue.cc:1035] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cd251d4d404145bea2cab1232e054b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:52:53.037736 2719 consensus_queue.cc:1035] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e303399dba40459a33885dc8871f14" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:52:53.061601 2717 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 00:52:54.652707 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:54.652992 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:52:54.656752 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814f83052ebe42dba4285b2be6ecf6f5: Sending DeleteTablet for 3 replicas of tablet 6b23d7977268431eaaf4397a7dfc35fe
I20260619 00:52:54.657459 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260619 00:52:54.657614 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35400
I20260619 00:52:54.657733 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b23d7977268431eaaf4397a7dfc35fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:52:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57528
I20260619 00:52:54.658339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2212
I20260619 00:52:54.660713 2794 tablet_replica.cc:330] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: stopping tablet replica
I20260619 00:52:54.660873 2793 tablet_replica.cc:330] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: stopping tablet replica
I20260619 00:52:54.660951 2794 raft_consensus.cc:2229] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:54.661052 2795 tablet_replica.cc:330] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: stopping tablet replica
I20260619 00:52:54.661047 2793 raft_consensus.cc:2229] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:52:54.661199 2794 raft_consensus.cc:2258] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:54.661278 2795 raft_consensus.cc:2229] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:52:54.661556 2793 raft_consensus.cc:2258] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:54.661588 2795 raft_consensus.cc:2258] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:52:54.662577 2794 ts_tablet_manager.cc:1905] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:54.662737 2793 ts_tablet_manager.cc:1905] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:54.662765 2795 ts_tablet_manager.cc:1905] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:52:54.669104 2794 ts_tablet_manager.cc:1918] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:54.669109 2793 ts_tablet_manager.cc:1918] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:54.669245 2793 log.cc:1196] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/6b23d7977268431eaaf4397a7dfc35fe
I20260619 00:52:54.669241 2794 log.cc:1196] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/6b23d7977268431eaaf4397a7dfc35fe
I20260619 00:52:54.669708 2793 ts_tablet_manager.cc:1939] T 6b23d7977268431eaaf4397a7dfc35fe P ccc9ae1782e2477e9a4f8860514c8913: Deleting consensus metadata
I20260619 00:52:54.669708 2794 ts_tablet_manager.cc:1939] T 6b23d7977268431eaaf4397a7dfc35fe P b2e303399dba40459a33885dc8871f14: Deleting consensus metadata
I20260619 00:52:54.670701 2795 ts_tablet_manager.cc:1918] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:52:54.670703 2125 catalog_manager.cc:4841] TS b2e303399dba40459a33885dc8871f14 (127.0.97.2:45887): tablet 6b23d7977268431eaaf4397a7dfc35fe (table test-table [id=dfbe855953ea43bd970760fad8f3961c]) successfully deleted
I20260619 00:52:54.670850 2795 log.cc:1196] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/6b23d7977268431eaaf4397a7dfc35fe
I20260619 00:52:54.671249 2125 catalog_manager.cc:4841] TS ccc9ae1782e2477e9a4f8860514c8913 (127.0.97.1:38719): tablet 6b23d7977268431eaaf4397a7dfc35fe (table test-table [id=dfbe855953ea43bd970760fad8f3961c]) successfully deleted
I20260619 00:52:54.671351 2795 ts_tablet_manager.cc:1939] T 6b23d7977268431eaaf4397a7dfc35fe P 38cd251d4d404145bea2cab1232e054b: Deleting consensus metadata
I20260619 00:52:54.672348 2128 catalog_manager.cc:4841] TS 38cd251d4d404145bea2cab1232e054b (127.0.97.3:41717): tablet 6b23d7977268431eaaf4397a7dfc35fe (table test-table [id=dfbe855953ea43bd970760fad8f3961c]) successfully deleted
Jun 19 00:52:54 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:52:54.686069 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:54.660084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.0.97.1:38719)
0619 00:52:54.671239 (+ 11155us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:54.671247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:54.671718 (+ 471us) server_negotiation.cc:407] Connection header received
0619 00:52:54.671902 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:54.671907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:54.671990 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:54.672122 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:54.673587 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.674299 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.675461 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.675847 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.675945 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:54.680261 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:54.680275 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:54.680290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:54.680324 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:54.683412 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.684190 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.684198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.684200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.684331 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.684647 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.684650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.684652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.685067 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:54.685307 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:54.685607 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 00:52:54.685776 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11007,"thread_start_us":10910,"threads_started":1}
W20260619 00:52:54.686787 2342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43114
W20260619 00:52:54.687515 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:54.687778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2212
I20260619 00:52:54.714617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2352
Jun 19 00:52:54 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:52:54.730397 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:54.715353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.0.97.2:45887)
0619 00:52:54.715900 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:54.715904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:54.715920 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:52:54.715976 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:54.715979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:54.716043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:54.716133 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:54.717642 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.718559 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.719999 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.720412 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.720512 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:54.724730 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:54.724746 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:54.724763 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:54.724799 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:54.727929 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.728566 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.728573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.728575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.728697 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.729112 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.729116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.729119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.729463 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:54.729699 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:54.729937 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 00:52:54.730131 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":324,"threads_started":1}
W20260619 00:52:54.731048 2482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48150
W20260619 00:52:54.731653 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2352: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:54.731812 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2352
I20260619 00:52:54.753845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2491
Jun 19 00:52:54 dist-test-slave-r7bh krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830372, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:52:54.769750 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:54.754617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.0.97.3:41717)
0619 00:52:54.755104 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:54.755109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:54.755124 (+ 15us) server_negotiation.cc:407] Connection header received
0619 00:52:54.755177 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:54.755181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:54.755239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:54.755330 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:54.756531 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.757376 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.758601 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:54.759026 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:54.759135 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:54.763594 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:54.763610 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:54.763627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:54.763665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:54.766775 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.767462 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.767470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.767473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.767593 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:54.767981 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:54.767988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:54.767991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:54.768446 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:54.768759 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:54.769087 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 00:52:54.769309 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260619 00:52:54.770664 2621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43454
W20260619 00:52:54.771401 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:52:54.771589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2491
I20260619 00:52:54.794505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2109
I20260619 00:52:54.795774 2181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:54.912161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2109
2026-06-19T00:52:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8238 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:55 dist-test-slave-r7bh krb5kdc[2803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:55 dist-test-slave-r7bh krb5kdc[2803](info): set up 2 sockets
Jun 19 00:52:55 dist-test-slave-r7bh krb5kdc[2803](info): commencing operation
krb5kdc: starting...
W20260619 00:52:56.009292 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:52:56 dist-test-slave-r7bh krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:56.198810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42631
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:56.516973 2816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:56.517279 2816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:56.517421 2816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:56.524736 2816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:52:56.524931 2816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:56.525010 2816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:56.525096 2816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:52:56.525162 2816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:52:56.535142 2816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:56.536550 2816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:56.537987 2816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:56.549867 2821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:56.549911 2822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:56.549817 2824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:56.550779 2816 server_base.cc:1034] running on GCE node
I20260619 00:52:56.551585 2816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:56.553068 2816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:56.554327 2816 hybrid_clock.cc:648] HybridClock initialized: now 1781830376554271 us; error 59 us; skew 500 ppm
I20260619 00:52:56.554704 2816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:56 dist-test-slave-r7bh krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:56.558174 2816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:52:56.559671 2816 webserver.cc:460] Webserver started at http://127.0.97.62:46317/ using document root <none> and password file <none>
I20260619 00:52:56.560690 2816 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:56.560843 2816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:56.561163 2816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:56.564072 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "c4ffb5a306d94840a89ce01a5f8b6770"
format_stamp: "Formatted at 2026-06-19 00:52:56 on dist-test-slave-r7bh"
server_key: "9db3c6369502f2c8fefe6868cf378547"
server_key_iv: "7b37ed18a5372fe6dd5a550f1c6b56f5"
server_key_version: "encryptionkey@0"
I20260619 00:52:56.564947 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "c4ffb5a306d94840a89ce01a5f8b6770"
format_stamp: "Formatted at 2026-06-19 00:52:56 on dist-test-slave-r7bh"
server_key: "9db3c6369502f2c8fefe6868cf378547"
server_key_iv: "7b37ed18a5372fe6dd5a550f1c6b56f5"
server_key_version: "encryptionkey@0"
I20260619 00:52:56.569859 2816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:52:56.573267 2831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:56.574532 2816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 00:52:56.574708 2816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "c4ffb5a306d94840a89ce01a5f8b6770"
format_stamp: "Formatted at 2026-06-19 00:52:56 on dist-test-slave-r7bh"
server_key: "9db3c6369502f2c8fefe6868cf378547"
server_key_iv: "7b37ed18a5372fe6dd5a550f1c6b56f5"
server_key_version: "encryptionkey@0"
I20260619 00:52:56.574895 2816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:56.604744 2816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:56.609295 2816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:56.609598 2816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:56.628099 2816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42631
I20260619 00:52:56.628122 2882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42631 every 8 connection(s)
I20260619 00:52:56.629576 2816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:52:56.633466 2883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:52:56.637135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2816
I20260619 00:52:56.637352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:52:56.637955 388 external_mini_cluster.cc:1428] Setting key b799ec1cbf28d8e2d4d44242e51daf6d
I20260619 00:52:56.644477 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770: Bootstrap starting.
I20260619 00:52:56.648083 2883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770: Neither blocks nor log segments found. Creating new log.
I20260619 00:52:56.649307 2883 log.cc:824] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770: Log is configured to *not* fsync() on all Append() calls
I20260619 00:52:56.652690 2883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770: No bootstrap required, opened a new log
Jun 19 00:52:56 dist-test-slave-r7bh krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:56.662846 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:56.639898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.97.62:42631)
0619 00:52:56.640660 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:56.640671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:56.640710 (+ 39us) server_negotiation.cc:407] Connection header received
0619 00:52:56.642892 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:56.642914 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:56.643308 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:56.643626 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:56.645470 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:56.646773 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:56.648541 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:56.649310 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:56.649496 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:56.654028 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:56.654038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:56.654055 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:56.654100 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:56.658115 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:56.658858 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:56.658872 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:56.658881 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:56.659095 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:56.659454 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:56.659459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:56.659462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:56.660198 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:56.660487 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:56.660911 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 00:52:56.661380 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":354,"threads_started":1}
I20260619 00:52:56.663515 2883 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } }
I20260619 00:52:56.663892 2883 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:52:56.664018 2883 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ffb5a306d94840a89ce01a5f8b6770, State: Initialized, Role: FOLLOWER
I20260619 00:52:56.664647 2883 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } }
I20260619 00:52:56.664913 2883 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:52:56.665057 2883 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:52:56.665254 2883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:52:56.667392 2883 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } }
I20260619 00:52:56.667874 2883 leader_election.cc:302] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ffb5a306d94840a89ce01a5f8b6770; no voters:
I20260619 00:52:56.668951 2883 leader_election.cc:288] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:52:56.669061 2888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:52:56.669842 2888 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [term 1 LEADER]: Becoming Leader. State: Replica: c4ffb5a306d94840a89ce01a5f8b6770, State: Running, Role: LEADER
I20260619 00:52:56.670447 2888 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } }
I20260619 00:52:56.671048 2883 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:52:56.673295 2889 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } } }
I20260619 00:52:56.673707 2889 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:56.673305 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ffb5a306d94840a89ce01a5f8b6770. Latest consensus state: current_term: 1 leader_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ffb5a306d94840a89ce01a5f8b6770" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42631 } } }
I20260619 00:52:56.674934 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770 [sys.catalog]: This master's current role is: LEADER
I20260619 00:52:56.679736 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:52:56.687822 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:52:56.697074 2893 catalog_manager.cc:1347] Generated new cluster ID: 1a5f8aaa562c47ebad97f13108ac1747
I20260619 00:52:56.697220 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:52:56.728847 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:52:56.730209 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:52:56.735795 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ffb5a306d94840a89ce01a5f8b6770: Generated new TSK 0
I20260619 00:52:56.736692 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:52:56.796437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:57.144122 2909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:57.144431 2909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:57.144580 2909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:57.152216 2909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:57.152343 2909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:57.152551 2909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:52:57.162779 2909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:57.164160 2909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:57.165729 2909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:57.180071 2914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:57.180015 2915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:57.179992 2917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:57.181021 2909 server_base.cc:1034] running on GCE node
I20260619 00:52:57.181716 2909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:57.182797 2909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:57.184130 2909 hybrid_clock.cc:648] HybridClock initialized: now 1781830377184072 us; error 44 us; skew 500 ppm
I20260619 00:52:57.184520 2909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:57 dist-test-slave-r7bh krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:57.187865 2909 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:52:57.189451 2909 webserver.cc:460] Webserver started at http://127.0.97.1:41403/ using document root <none> and password file <none>
I20260619 00:52:57.190181 2909 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:57.190330 2909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:57.190655 2909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:57.193746 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "781332d299d64d60a1b6b9918b33bf0e"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "22078d9bd7ce4b93be9c42a64c6a691b"
server_key_iv: "8cf51108a69b596cb57e0750a6b69204"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.194620 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "781332d299d64d60a1b6b9918b33bf0e"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "22078d9bd7ce4b93be9c42a64c6a691b"
server_key_iv: "8cf51108a69b596cb57e0750a6b69204"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.199879 2909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:52:57.203540 2924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.204964 2909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:57.205144 2909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "781332d299d64d60a1b6b9918b33bf0e"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "22078d9bd7ce4b93be9c42a64c6a691b"
server_key_iv: "8cf51108a69b596cb57e0750a6b69204"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.205350 2909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:57.216219 2909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:57.220147 2909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:57.220449 2909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:57.221432 2909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:57.222707 2909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:57.222810 2909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.222965 2909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:57.223043 2909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.256666 2909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44297
I20260619 00:52:57.256731 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44297 every 8 connection(s)
I20260619 00:52:57.258180 2909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:52:57.267489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 2909
I20260619 00:52:57.267735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:52:57.268383 388 external_mini_cluster.cc:1428] Setting key 082da7b1fde461b994b6688c66404331
Jun 19 00:52:57 dist-test-slave-r7bh krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:57.281332 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:57.260882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45193 (local address 127.0.97.62:42631)
0619 00:52:57.261317 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:57.261322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:57.262486 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 00:52:57.264510 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:57.264517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:57.264618 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:57.264768 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:57.266547 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.267312 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.269472 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.269933 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.270089 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:57.274660 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:57.274666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:57.274669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:57.274703 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:57.277409 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:57.278544 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:57.278548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:57.278550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:57.278715 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:57.279297 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:57.279301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:57.279303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:57.279689 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:57.279812 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:57.280873 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0619 00:52:57.281079 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260619 00:52:57.283067 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:57.261440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42631 (local address 127.0.97.1:45193)
0619 00:52:57.262258 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:57.262301 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:57.264170 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:57.265057 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:57.265066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:57.265660 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:57.266279 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:57.266297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.267507 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.267514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:57.269269 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:57.269296 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.270190 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.270200 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:57.270564 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:57.271415 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:57.271455 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:57.274386 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:57.277604 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:57.277618 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:57.277642 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:57.278352 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:57.278849 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:57.278857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:57.278864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:57.279141 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:57.279817 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:57.279836 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:57.280333 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 00:52:57.280679 (+ 346us) client_negotiation.cc:240] Negotiation successful
0619 00:52:57.281364 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":317,"threads_started":1}
I20260619 00:52:57.285055 3037 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42631
I20260619 00:52:57.285403 3037 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:57.286146 3037 heartbeater.cc:511] Master 127.0.97.62:42631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:52:57.288335 2848 ts_manager.cc:194] Registered new tserver with Master: 781332d299d64d60a1b6b9918b33bf0e (127.0.97.1:44297)
I20260619 00:52:57.289908 2848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:52:57.312777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:57.649358 3043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:57.649688 3043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:57.649852 3043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:57.657419 3043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:57.657569 3043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:57.657753 3043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:52:57.667961 3043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:57.669332 3043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:57.670728 3043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:57.684758 3048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:57.684897 3049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:57.684868 3051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:57.685894 3043 server_base.cc:1034] running on GCE node
I20260619 00:52:57.686523 3043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:57.687479 3043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:57.688699 3043 hybrid_clock.cc:648] HybridClock initialized: now 1781830377688664 us; error 31 us; skew 500 ppm
I20260619 00:52:57.689055 3043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:57 dist-test-slave-r7bh krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:57.692147 3043 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:52:57.693696 3043 webserver.cc:460] Webserver started at http://127.0.97.2:37115/ using document root <none> and password file <none>
I20260619 00:52:57.694407 3043 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:57.694543 3043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:57.694877 3043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:57.697944 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "577cdd76a5154f5bb4c838945340d014"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "0f1c0173e22e802efea14f13b1b1c946"
server_key_iv: "3f42d1bb8dfe8bcb01d7403cc07ba1ad"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.698748 3043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "577cdd76a5154f5bb4c838945340d014"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "0f1c0173e22e802efea14f13b1b1c946"
server_key_iv: "3f42d1bb8dfe8bcb01d7403cc07ba1ad"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.704325 3043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 00:52:57.708174 3058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.709822 3043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:52:57.710043 3043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "577cdd76a5154f5bb4c838945340d014"
format_stamp: "Formatted at 2026-06-19 00:52:57 on dist-test-slave-r7bh"
server_key: "0f1c0173e22e802efea14f13b1b1c946"
server_key_iv: "3f42d1bb8dfe8bcb01d7403cc07ba1ad"
server_key_version: "encryptionkey@0"
I20260619 00:52:57.710271 3043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:57.734514 3043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:57.739275 3043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:57.739596 3043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:57.740643 3043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:57.742128 3043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:57.742239 3043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.742390 3043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:57.742462 3043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:57.775542 3043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38221
I20260619 00:52:57.775588 3170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38221 every 8 connection(s)
I20260619 00:52:57.776842 3043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:52:57.781911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3043
I20260619 00:52:57.782112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:52:57.782563 388 external_mini_cluster.cc:1428] Setting key 25362b59c804aa04d48b65399b9be36c
Jun 19 00:52:57 dist-test-slave-r7bh krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:57.798178 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:57.779526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49217 (local address 127.0.97.62:42631)
0619 00:52:57.779697 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:57.779702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:57.780983 (+ 1281us) server_negotiation.cc:407] Connection header received
0619 00:52:57.783156 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:57.783161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:57.783255 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:57.783381 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:57.785422 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.786638 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.788221 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.789011 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.789154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:57.792581 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:57.792586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:57.792590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:57.792620 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:57.795157 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:57.795860 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:57.795866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:57.795869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:57.796003 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:57.796462 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:57.796469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:57.796472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:57.796732 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:57.796928 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:57.797587 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 00:52:57.797800 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:52:57.799165 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:57.780096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42631 (local address 127.0.97.2:49217)
0619 00:52:57.780777 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:57.780871 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:57.782832 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:57.783650 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:57.783661 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:57.784507 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:57.785230 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:57.785252 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.786813 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.786817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:57.788071 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:57.788083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:57.789237 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:57.789242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:57.789467 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:57.790222 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:57.790249 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:57.792349 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:57.795295 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:57.795305 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:57.795321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:57.795737 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:57.796111 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:57.796117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:57.796121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:57.796343 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:57.796877 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:57.796886 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:57.797235 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 00:52:57.797507 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 00:52:57.798019 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:52:57.800738 3171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42631
I20260619 00:52:57.801155 3171 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:57.801965 3171 heartbeater.cc:511] Master 127.0.97.62:42631 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:52:57.803525 2848 ts_manager.cc:194] Registered new tserver with Master: 577cdd76a5154f5bb4c838945340d014 (127.0.97.2:38221)
I20260619 00:52:57.804317 2848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:52:57.823818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:52:58.145844 3176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:52:58.146165 3176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:52:58.146267 3176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:52:58.153753 3176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:52:58.153898 3176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:52:58.154044 3176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:52:58.164103 3176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:52:58.165418 3176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:52:58.166715 3176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:52:58.181284 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:58.181257 3182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:52:58.181785 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:52:58.182186 3176 server_base.cc:1034] running on GCE node
I20260619 00:52:58.182933 3176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:52:58.183890 3176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:52:58.185079 3176 hybrid_clock.cc:648] HybridClock initialized: now 1781830378185015 us; error 61 us; skew 500 ppm
I20260619 00:52:58.185385 3176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:52:58 dist-test-slave-r7bh krb5kdc[2803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:52:58.188524 3176 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:52:58.189905 3176 webserver.cc:460] Webserver started at http://127.0.97.3:35201/ using document root <none> and password file <none>
I20260619 00:52:58.190542 3176 fs_manager.cc:359] Metadata directory not provided
I20260619 00:52:58.190639 3176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:52:58.191010 3176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:52:58.193936 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "d36768dd22644bccad49cad55a935042"
format_stamp: "Formatted at 2026-06-19 00:52:58 on dist-test-slave-r7bh"
server_key: "3b28494ac21c417a4a46093ca1c14d21"
server_key_iv: "2e53168b6b0c7177d708243a5b201e8d"
server_key_version: "encryptionkey@0"
I20260619 00:52:58.194703 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d36768dd22644bccad49cad55a935042"
format_stamp: "Formatted at 2026-06-19 00:52:58 on dist-test-slave-r7bh"
server_key: "3b28494ac21c417a4a46093ca1c14d21"
server_key_iv: "2e53168b6b0c7177d708243a5b201e8d"
server_key_version: "encryptionkey@0"
I20260619 00:52:58.199715 3176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:52:58.203209 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:58.204548 3176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:52:58.204705 3176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "d36768dd22644bccad49cad55a935042"
format_stamp: "Formatted at 2026-06-19 00:52:58 on dist-test-slave-r7bh"
server_key: "3b28494ac21c417a4a46093ca1c14d21"
server_key_iv: "2e53168b6b0c7177d708243a5b201e8d"
server_key_version: "encryptionkey@0"
I20260619 00:52:58.204957 3176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:52:58.229575 3176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:52:58.233946 3176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:52:58.234246 3176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:52:58.235325 3176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:52:58.236665 3176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:52:58.236738 3176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:58.236894 3176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:52:58.236944 3176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:52:58.273047 3176 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39787
I20260619 00:52:58.273108 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39787 every 8 connection(s)
I20260619 00:52:58.274431 3176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:52:58.282199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3176
I20260619 00:52:58.282420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:52:58.282899 388 external_mini_cluster.cc:1428] Setting key 11026360e8366b50606c23168beb670b
Jun 19 00:52:58 dist-test-slave-r7bh krb5kdc[2803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:52:58.292990 3037 heartbeater.cc:503] Master 127.0.97.62:42631 was elected leader, sending a full tablet report...
I20260619 00:52:58.295187 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:58.277230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60673 (local address 127.0.97.62:42631)
0619 00:52:58.277413 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:58.277418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:58.278834 (+ 1416us) server_negotiation.cc:407] Connection header received
0619 00:52:58.280739 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:58.280749 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:58.280885 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:58.280986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:58.282756 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:58.283626 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:58.285591 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:52:58.286026 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:52:58.286145 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:58.289644 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:52:58.289648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:52:58.289652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:52:58.289684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:52:58.292003 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:58.292695 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:58.292699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:58.292701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:58.292873 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:52:58.293366 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:52:58.293371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:52:58.293374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:52:58.293721 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:52:58.293885 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 00:52:58.294611 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 00:52:58.294828 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:52:58.296175 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:52:58.277763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42631 (local address 127.0.97.3:60673)
0619 00:52:58.278671 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 00:52:58.278717 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:52:58.280463 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:52:58.281206 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:52:58.281214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:52:58.281743 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:52:58.282569 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:58.282602 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:58.283779 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:58.283785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:58.285360 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:52:58.285374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:52:58.286236 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:52:58.286241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:52:58.286436 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:52:58.287180 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:52:58.287202 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:52:58.289417 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:52:58.292139 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:58.292146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:58.292159 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:58.292579 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:58.292988 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:52:58.292993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:52:58.292997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:52:58.293239 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:52:58.293866 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:52:58.293874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:52:58.294230 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 00:52:58.294504 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 00:52:58.295054 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":386,"threads_started":1}
I20260619 00:52:58.297742 3304 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42631
I20260619 00:52:58.298074 3304 heartbeater.cc:464] Registering TS with master...
I20260619 00:52:58.298843 3304 heartbeater.cc:511] Master 127.0.97.62:42631 requested a full tablet report, sending...
I20260619 00:52:58.300321 2847 ts_manager.cc:194] Registered new tserver with Master: d36768dd22644bccad49cad55a935042 (127.0.97.3:39787)
I20260619 00:52:58.301239 2847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60673
I20260619 00:52:58.311400 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 00:52:58.342262 3039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:52:58.339671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.97.62:42631)
0619 00:52:58.339874 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 00:52:58.339879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:52:58.340433 (+ 554us) server_negotiation.cc:407] Connection header received
0619 00:52:58.340553 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:52:58.340557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:52:58.340624 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:52:58.340724 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:52:58.342072 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51988: BlockingRecv error: recv got EOF from 127.0.0.1:51988 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:52:58.345599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2909
W20260619 00:52:58.348475 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2909: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44297: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:52:58.348639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2909
I20260619 00:52:58.364593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3043
W20260619 00:52:58.367344 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3043: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38221: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:52:58.367549 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3043
I20260619 00:52:58.383303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3176
W20260619 00:52:58.386270 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3176: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39787: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:52:58.386437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3176
I20260619 00:52:58.401726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 2816
I20260619 00:52:58.403067 2878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:52:58.505249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 2816
2026-06-19T00:52:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3590 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:52:58 dist-test-slave-r7bh krb5kdc[3321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:52:58 dist-test-slave-r7bh krb5kdc[3321](info): set up 2 sockets
Jun 19 00:52:58 dist-test-slave-r7bh krb5kdc[3321](info): commencing operation
krb5kdc: starting...
W20260619 00:52:59.617492 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:52:59 dist-test-slave-r7bh krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:52:59.805282 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:00.127511 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:00.127866 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:00.128043 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:00.135188 3334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:00.135346 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:00.135457 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:00.135560 3334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:00.135653 3334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:00.146153 3334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:00.147500 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:00.148826 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:00.160925 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:00.160871 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:00.160843 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:00.161805 3334 server_base.cc:1034] running on GCE node
I20260619 00:53:00.162640 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:00.164139 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:00.165374 3334 hybrid_clock.cc:648] HybridClock initialized: now 1781830380165341 us; error 35 us; skew 500 ppm
I20260619 00:53:00.165772 3334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:00 dist-test-slave-r7bh krb5kdc[3321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:00.169129 3334 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:00.170770 3334 webserver.cc:460] Webserver started at http://127.0.97.62:34425/ using document root <none> and password file <none>
I20260619 00:53:00.171520 3334 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:00.171667 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:00.171972 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:00.175191 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2"
format_stamp: "Formatted at 2026-06-19 00:53:00 on dist-test-slave-r7bh"
server_key: "3c4c46460f6c955a475b43e4b9a069dc"
server_key_iv: "9a60eb1482688f41128ee7c887d031a7"
server_key_version: "encryptionkey@0"
I20260619 00:53:00.176203 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2"
format_stamp: "Formatted at 2026-06-19 00:53:00 on dist-test-slave-r7bh"
server_key: "3c4c46460f6c955a475b43e4b9a069dc"
server_key_iv: "9a60eb1482688f41128ee7c887d031a7"
server_key_version: "encryptionkey@0"
I20260619 00:53:00.183131 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260619 00:53:00.187184 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:00.188771 3334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:00.189064 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2"
format_stamp: "Formatted at 2026-06-19 00:53:00 on dist-test-slave-r7bh"
server_key: "3c4c46460f6c955a475b43e4b9a069dc"
server_key_iv: "9a60eb1482688f41128ee7c887d031a7"
server_key_version: "encryptionkey@0"
I20260619 00:53:00.189288 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:00.223435 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:00.228147 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:00.228469 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:00.248688 3334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43433
I20260619 00:53:00.248709 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43433 every 8 connection(s)
I20260619 00:53:00.250216 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:00.254207 3401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:00.254379 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3334
I20260619 00:53:00.254580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:00.255365 388 external_mini_cluster.cc:1428] Setting key 16666c6c2546bf706d7169ce938a43f6
I20260619 00:53:00.268213 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: Bootstrap starting.
Jun 19 00:53:00 dist-test-slave-r7bh krb5kdc[3321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:00.272101 3401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:00.273279 3401 log.cc:824] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:00.276312 3401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: No bootstrap required, opened a new log
I20260619 00:53:00.279975 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:00.257374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.0.97.62:43433)
0619 00:53:00.258270 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:00.258282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:00.258328 (+ 46us) server_negotiation.cc:407] Connection header received
0619 00:53:00.260357 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:00.260382 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:00.260799 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:00.261242 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:00.262987 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:00.264277 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:00.266102 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:00.266865 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:00.267003 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:00.271725 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:00.271734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:00.271752 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:00.271797 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:00.275079 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:00.275920 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:00.275932 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:00.275938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:00.276117 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:00.276562 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:00.276566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:00.276570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:00.277356 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:00.277663 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:00.278088 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 00:53:00.278574 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":417,"threads_started":1}
I20260619 00:53:00.283926 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:53:00.286037 3401 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 00:53:00.286499 3401 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:00.286608 3401 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b32a9eb3a7429b81c383ee1cfa4be2, State: Initialized, Role: FOLLOWER
I20260619 00:53:00.287292 3401 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 00:53:00.287606 3401 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:00.287755 3401 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:00.287937 3401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:00.290468 3401 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 00:53:00.290977 3401 leader_election.cc:302] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b32a9eb3a7429b81c383ee1cfa4be2; no voters:
I20260619 00:53:00.291589 3401 leader_election.cc:288] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:00.291787 3407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:00.292790 3407 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [term 1 LEADER]: Becoming Leader. State: Replica: d9b32a9eb3a7429b81c383ee1cfa4be2, State: Running, Role: LEADER
I20260619 00:53:00.293428 3407 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } }
I20260619 00:53:00.293802 3401 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:00.302273 3409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b32a9eb3a7429b81c383ee1cfa4be2. Latest consensus state: current_term: 1 leader_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 00:53:00.302454 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b32a9eb3a7429b81c383ee1cfa4be2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43433 } } }
I20260619 00:53:00.302780 3409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:00.302814 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2 [sys.catalog]: This master's current role is: LEADER
W20260619 00:53:00.317150 3421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:53:00.317358 3421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:53:00.317546 3422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:00.319823 3422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:00.329162 3422 catalog_manager.cc:1347] Generated new cluster ID: eb9ec494e0764defbb041a2103607c46
I20260619 00:53:00.329370 3422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:00.335685 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:00.324656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.0.97.62:43433)
0619 00:53:00.324851 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:00.324858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:00.326825 (+ 1967us) server_negotiation.cc:407] Connection header received
0619 00:53:00.326991 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:00.326997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:00.327073 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:00.327200 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:00.328750 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:00.329951 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:00.331810 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:00.332572 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:00.332713 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:00.333921 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:00.333929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:00.333934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:00.334763 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:00.334858 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:00.334889 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:00.335012 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:00.335158 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 00:53:00.335270 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:53:00.340161 3422 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:00.342092 3422 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:00.357908 3422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b32a9eb3a7429b81c383ee1cfa4be2: Generated new TSK 0
I20260619 00:53:00.359036 3422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:53:00.372282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3334
I20260619 00:53:00.374166 3396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:00.464581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3334
2026-06-19T00:53:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1950 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:00 dist-test-slave-r7bh krb5kdc[3435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:00 dist-test-slave-r7bh krb5kdc[3435](info): set up 2 sockets
Jun 19 00:53:00 dist-test-slave-r7bh krb5kdc[3435](info): commencing operation
krb5kdc: starting...
W20260619 00:53:01.553328 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:01 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:01.752707 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44143
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:02.100615 3448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:02.101022 3448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:02.101140 3448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:02.108634 3448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:02.108768 3448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:02.108897 3448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:02.108978 3448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:02.109020 3448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:02.119320 3448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:02.120693 3448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:02.122164 3448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:02.134775 3453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:02.134735 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:02.134693 3454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:02.135957 3448 server_base.cc:1034] running on GCE node
I20260619 00:53:02.136912 3448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:02.138449 3448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:02.139705 3448 hybrid_clock.cc:648] HybridClock initialized: now 1781830382139665 us; error 65 us; skew 500 ppm
I20260619 00:53:02.140113 3448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:02 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:02.143463 3448 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:02.145320 3448 webserver.cc:460] Webserver started at http://127.0.97.62:45117/ using document root <none> and password file <none>
I20260619 00:53:02.146078 3448 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:02.146239 3448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:02.146564 3448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:02.149792 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "c0892e220ded4c31916cd26dd7774173"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "6217f2d4fc980ac4deebc5057305a3a5"
server_key_iv: "3f9222aae7eb9d87326a866088b3c71f"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.150609 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0892e220ded4c31916cd26dd7774173"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "6217f2d4fc980ac4deebc5057305a3a5"
server_key_iv: "3f9222aae7eb9d87326a866088b3c71f"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.156059 3448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:02.159835 3463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:02.161367 3448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:02.161556 3448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "c0892e220ded4c31916cd26dd7774173"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "6217f2d4fc980ac4deebc5057305a3a5"
server_key_iv: "3f9222aae7eb9d87326a866088b3c71f"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.161757 3448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:02.189184 3448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:02.197449 3448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:02.197779 3448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:02.216696 3448 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44143
I20260619 00:53:02.216717 3514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44143 every 8 connection(s)
I20260619 00:53:02.218240 3448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:02.222136 3515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:02.223951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3448
I20260619 00:53:02.224164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:02.224915 388 external_mini_cluster.cc:1428] Setting key 483dd8fed6b220eef4c1ef2f592f898f
I20260619 00:53:02.235237 3515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173: Bootstrap starting.
I20260619 00:53:02.238875 3515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:02.239923 3515 log.cc:824] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:53:02 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:02.243898 3515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173: No bootstrap required, opened a new log
I20260619 00:53:02.250799 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:02.226942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.97.62:44143)
0619 00:53:02.227764 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:02.227781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:02.227836 (+ 55us) server_negotiation.cc:407] Connection header received
0619 00:53:02.230171 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:02.230198 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:02.230695 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:02.231124 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:02.232339 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.234359 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.235759 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.236745 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.237009 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:02.241433 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:02.241446 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:02.241469 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:02.241518 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:02.245635 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:02.246512 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:02.246524 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:02.246537 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:02.246771 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:02.247248 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:02.247253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:02.247258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:02.248015 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:02.248361 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:02.248764 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 00:53:02.249270 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
I20260619 00:53:02.254298 3515 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260619 00:53:02.254709 3515 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:02.254782 3515 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0892e220ded4c31916cd26dd7774173, State: Initialized, Role: FOLLOWER
I20260619 00:53:02.255301 3515 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260619 00:53:02.255564 3515 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:02.255661 3515 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:02.255795 3515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:02.258507 3515 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260619 00:53:02.259074 3515 leader_election.cc:302] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0892e220ded4c31916cd26dd7774173; no voters:
I20260619 00:53:02.259658 3515 leader_election.cc:288] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:02.259819 3520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:02.261135 3520 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [term 1 LEADER]: Becoming Leader. State: Replica: c0892e220ded4c31916cd26dd7774173, State: Running, Role: LEADER
I20260619 00:53:02.261694 3520 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } }
I20260619 00:53:02.262054 3515 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:02.264842 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0892e220ded4c31916cd26dd7774173. Latest consensus state: current_term: 1 leader_uuid: "c0892e220ded4c31916cd26dd7774173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } } }
I20260619 00:53:02.265234 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:02.265750 3521 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0892e220ded4c31916cd26dd7774173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0892e220ded4c31916cd26dd7774173" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44143 } } }
I20260619 00:53:02.266391 3521 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:02.268179 3528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:02.273730 3528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:02.283922 3528 catalog_manager.cc:1347] Generated new cluster ID: 696c871ffbb247799b6b1aa7ab2066c5
I20260619 00:53:02.284080 3528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:02.298904 3528 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:02.300246 3528 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:02.307356 3528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0892e220ded4c31916cd26dd7774173: Generated new TSK 0
I20260619 00:53:02.308287 3528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:53:02.356491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:02.694535 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:02.694871 3541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:02.694973 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:02.702797 3541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:02.702936 3541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:02.703061 3541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:02.713002 3541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:02.714422 3541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:02.715962 3541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:02.729754 3549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:02.729944 3547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:02.730122 3546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:02.730640 3541 server_base.cc:1034] running on GCE node
I20260619 00:53:02.731264 3541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:02.732264 3541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:02.733505 3541 hybrid_clock.cc:648] HybridClock initialized: now 1781830382733430 us; error 76 us; skew 500 ppm
I20260619 00:53:02.733887 3541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:02 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:02.741158 3541 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:53:02.743088 3541 webserver.cc:460] Webserver started at http://127.0.97.1:43089/ using document root <none> and password file <none>
I20260619 00:53:02.743896 3541 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:02.744019 3541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:02.744313 3541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:02.747768 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "bfebbaa33b864b7b987eb791079dba28"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "97ab87f53cfac447a2f4d86cd9af338d"
server_key_iv: "9b29f38cb8738e743c8f9b6bdb88f6ad"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.748651 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bfebbaa33b864b7b987eb791079dba28"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "97ab87f53cfac447a2f4d86cd9af338d"
server_key_iv: "9b29f38cb8738e743c8f9b6bdb88f6ad"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.755009 3541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 00:53:02.759290 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:02.761241 3541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 00:53:02.761475 3541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bfebbaa33b864b7b987eb791079dba28"
format_stamp: "Formatted at 2026-06-19 00:53:02 on dist-test-slave-r7bh"
server_key: "97ab87f53cfac447a2f4d86cd9af338d"
server_key_iv: "9b29f38cb8738e743c8f9b6bdb88f6ad"
server_key_version: "encryptionkey@0"
I20260619 00:53:02.761744 3541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:02.793495 3541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:02.798056 3541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:02.798374 3541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:02.799393 3541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:02.800876 3541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:02.800963 3541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:02.801127 3541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:02.801190 3541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:02.836133 3541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39973
I20260619 00:53:02.836230 3668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39973 every 8 connection(s)
I20260619 00:53:02.837507 3541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:02.847601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3541
I20260619 00:53:02.847818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:02.848500 388 external_mini_cluster.cc:1428] Setting key bd81addf16d0ee6d88def246f38519a7
Jun 19 00:53:02 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:02.858525 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:02.840198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43541 (local address 127.0.97.62:44143)
0619 00:53:02.840697 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:02.840702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:02.841888 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 00:53:02.843771 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:02.843776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:02.843865 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:02.844008 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:02.845637 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.846379 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.848120 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.848586 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.848724 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:02.852756 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:02.852761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:02.852764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:02.852795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:02.855344 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:02.856098 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:02.856104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:02.856107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:02.856225 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:02.856653 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:02.856658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:02.856661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:02.856991 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:02.857140 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:02.857936 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 00:53:02.858166 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260619 00:53:02.859539 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:02.840926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.1:43541)
0619 00:53:02.841712 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:02.841757 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:02.843478 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:02.844248 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:02.844257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:02.844802 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:02.845456 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:02.845482 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.846553 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.846557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:02.847963 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:02.847981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:02.848899 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:02.848907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:02.849210 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:02.849976 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:02.849999 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:02.852536 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:02.855524 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:02.855533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:02.855550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:02.855972 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:02.856334 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:02.856340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:02.856344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:02.856536 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:02.857112 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:02.857120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:02.857484 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 00:53:02.857794 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 00:53:02.858357 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":326,"threads_started":1}
I20260619 00:53:02.861289 3669 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260619 00:53:02.861647 3669 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:02.862577 3669 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260619 00:53:02.864799 3480 ts_manager.cc:194] Registered new tserver with Master: bfebbaa33b864b7b987eb791079dba28 (127.0.97.1:39973)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:53:02.866585 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:53:02.892794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:03.224617 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:03.224988 3675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:03.225198 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:03.232506 3675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:03.232604 3675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:03.232841 3675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:03.242659 3675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:03.243975 3675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:03.245384 3675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:03.258756 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:03.258709 3683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:03.258845 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:03.259828 3675 server_base.cc:1034] running on GCE node
I20260619 00:53:03.260509 3675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:03.261598 3675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:03.262830 3675 hybrid_clock.cc:648] HybridClock initialized: now 1781830383262749 us; error 73 us; skew 500 ppm
I20260619 00:53:03.263181 3675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:03.266516 3675 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:53:03.268009 3675 webserver.cc:460] Webserver started at http://127.0.97.2:46741/ using document root <none> and password file <none>
I20260619 00:53:03.268661 3675 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:03.268754 3675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:03.269061 3675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:03.272235 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "5aa5cee3513947fbbe84e47596516341"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "91df9e39d2ed8fe6d338b83bb6a486f2"
server_key_iv: "5586ae54f0b0e303d23763017ff292d8"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.273031 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5aa5cee3513947fbbe84e47596516341"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "91df9e39d2ed8fe6d338b83bb6a486f2"
server_key_iv: "5586ae54f0b0e303d23763017ff292d8"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.278061 3675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:03.281733 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.283245 3675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:03.283393 3675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "5aa5cee3513947fbbe84e47596516341"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "91df9e39d2ed8fe6d338b83bb6a486f2"
server_key_iv: "5586ae54f0b0e303d23763017ff292d8"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.283582 3675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:03.298756 3675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:03.303037 3675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:03.303287 3675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:03.304171 3675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:03.305485 3675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:03.305574 3675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.305657 3675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:03.305703 3675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.338825 3675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33965
I20260619 00:53:03.338955 3802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33965 every 8 connection(s)
I20260619 00:53:03.340162 3675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:03.342260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3675
I20260619 00:53:03.342454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:03.343007 388 external_mini_cluster.cc:1428] Setting key bbf5b413f8c7a5ccf91292119c8eacd8
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:53:03.362668 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.342916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55291 (local address 127.0.97.62:44143)
0619 00:53:03.343132 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.343140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.344561 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 00:53:03.347294 (+ 2733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.347298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.347364 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.347447 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.349578 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.350288 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.352029 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.352450 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.352587 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.356489 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:03.356494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:03.356497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:03.356530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:03.359183 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.360149 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.360155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.360160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.360318 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.360865 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.360873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.360875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.361143 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:03.361252 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:03.362179 (+ 927us) server_negotiation.cc:299] Negotiation successful
0619 00:53:03.362347 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:53:03.363840 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.343528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.2:55291)
0619 00:53:03.344366 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:03.344435 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:03.346985 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:03.347697 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:03.347713 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:03.348511 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:03.349387 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:03.349426 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.350446 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.350454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:03.351873 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:03.351885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.352678 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.352684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:03.352986 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.353697 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:03.353720 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:03.356225 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:03.359331 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:03.359343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:03.359366 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:03.360005 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:03.360436 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:03.360443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:03.360448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:03.360709 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:03.361282 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:03.361292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:03.361757 (+ 465us) client_negotiation.cc:769] Sending connection context
0619 00:53:03.362084 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 00:53:03.362503 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":371,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:53:03.365446 3803 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260619 00:53:03.365800 3803 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:03.366730 3803 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260619 00:53:03.368242 3480 ts_manager.cc:194] Registered new tserver with Master: 5aa5cee3513947fbbe84e47596516341 (127.0.97.2:33965)
I20260619 00:53:03.369197 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:53:03.389269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:03.715135 3808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:03.715415 3808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:03.715569 3808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:03.722723 3808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:03.722842 3808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:03.723037 3808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:03.733253 3808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:03.734666 3808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:03.736104 3808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:03.749377 3813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:03.749414 3814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:03.749519 3816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:03.750648 3808 server_base.cc:1034] running on GCE node
I20260619 00:53:03.751268 3808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:03.752261 3808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:03.753446 3808 hybrid_clock.cc:648] HybridClock initialized: now 1781830383753389 us; error 51 us; skew 500 ppm
I20260619 00:53:03.753780 3808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:03.757046 3808 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:53:03.758565 3808 webserver.cc:460] Webserver started at http://127.0.97.3:43275/ using document root <none> and password file <none>
I20260619 00:53:03.759224 3808 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:03.759318 3808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:03.759567 3808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:03.762583 3808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "91566ebae6624b78af056406539727af"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "8fa19c681de0e6672c08c3e6a6ed0761"
server_key_iv: "cf0974fbf10b897fa0d18715d1cfda6c"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.763402 3808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91566ebae6624b78af056406539727af"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "8fa19c681de0e6672c08c3e6a6ed0761"
server_key_iv: "cf0974fbf10b897fa0d18715d1cfda6c"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.769034 3808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:53:03.772959 3823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.774590 3808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:03.774796 3808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "91566ebae6624b78af056406539727af"
format_stamp: "Formatted at 2026-06-19 00:53:03 on dist-test-slave-r7bh"
server_key: "8fa19c681de0e6672c08c3e6a6ed0761"
server_key_iv: "cf0974fbf10b897fa0d18715d1cfda6c"
server_key_version: "encryptionkey@0"
I20260619 00:53:03.775002 3808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:03.800746 3808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:03.805220 3808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:03.805518 3808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:03.806489 3808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:03.807904 3808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:03.807976 3808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.808099 3808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:03.808147 3808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:03.841754 3808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44307
I20260619 00:53:03.841789 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44307 every 8 connection(s)
I20260619 00:53:03.842999 3808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:03.848455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3808
I20260619 00:53:03.848650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:03.849205 388 external_mini_cluster.cc:1428] Setting key a58bb64237cacc4d0622e9cc8cc72d4b
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:03.863359 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.845467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38689 (local address 127.0.97.62:44143)
0619 00:53:03.845679 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.845685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.846795 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 00:53:03.848784 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.848790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.848900 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.849009 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.850907 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.851728 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.853378 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.853766 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.853899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.857390 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:03.857395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:03.857398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:03.857429 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:03.860063 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.860751 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.860755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.860757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.860986 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.861389 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.861393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.861395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.861753 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:03.861878 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:03.862734 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 00:53:03.862997 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:53:03.864557 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.845943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44143 (local address 127.0.97.3:38689)
0619 00:53:03.846631 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:03.846672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:03.848408 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:03.849277 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:03.849288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:03.849911 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:03.850713 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:03.850739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.851883 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.851890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:03.853224 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:03.853235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.854007 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.854014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:03.854234 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.854941 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:03.854964 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:03.857190 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:03.860184 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:03.860191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:03.860203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:03.860609 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:03.861084 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:03.861087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:03.861090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:03.861262 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:03.861873 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:03.861878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:03.862242 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 00:53:03.862588 (+ 346us) client_negotiation.cc:240] Negotiation successful
0619 00:53:03.863141 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":276,"threads_started":1}
I20260619 00:53:03.866144 3936 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44143
I20260619 00:53:03.866503 3936 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:03.867281 3936 heartbeater.cc:511] Master 127.0.97.62:44143 requested a full tablet report, sending...
I20260619 00:53:03.868710 3480 ts_manager.cc:194] Registered new tserver with Master: 91566ebae6624b78af056406539727af (127.0.97.3:44307)
I20260619 00:53:03.869539 3480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38689
I20260619 00:53:03.870030 3669 heartbeater.cc:503] Master 127.0.97.62:44143 was elected leader, sending a full tablet report...
I20260619 00:53:03.876935 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:03.924427 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.911072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.97.62:44143)
0619 00:53:03.911243 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.911248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.911776 (+ 528us) server_negotiation.cc:407] Connection header received
0619 00:53:03.911929 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.911933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.911996 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.912092 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.913531 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.914260 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.915408 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.915794 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.915883 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.919826 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:03.919831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:03.919835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:03.919866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:03.922221 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.922803 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.922806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.922809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.922920 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.923273 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.923277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.923279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.923567 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:03.923751 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:03.924025 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 00:53:03.924197 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 00:53:03.925032 3480 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60460
I20260619 00:53:03.926779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3541
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:03.942327 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.927806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.97.1:39973)
0619 00:53:03.928416 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.928423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.928447 (+ 24us) server_negotiation.cc:407] Connection header received
0619 00:53:03.928499 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.928506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.928728 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.928919 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.930091 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.931073 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.932248 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.932849 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.932988 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.936920 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:03.936934 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:03.936953 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:03.936996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:03.939761 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.940453 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.940460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.940462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.940604 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.941021 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.941024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.941026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.941358 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:03.941578 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:03.941846 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 00:53:03.942047 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":304,"threads_started":1}
W20260619 00:53:03.943706 3664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52388
W20260619 00:53:03.944872 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:53:03.945091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3541
I20260619 00:53:03.960742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3675
Jun 19 00:53:03 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:53:03.977710 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.961751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.0.97.2:33965)
0619 00:53:03.962288 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.962295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.962321 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:53:03.962398 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.962404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.962644 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.962772 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.964292 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.965753 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.967193 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:03.967757 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:03.967906 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:03.972227 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:03.972238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:03.972258 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:03.972294 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:03.975112 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.975728 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.975736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.975738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.975865 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:03.976258 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:03.976262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:03.976265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:03.976605 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:03.976887 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:03.977214 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 00:53:03.977418 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":288,"threads_started":1}
W20260619 00:53:03.979149 3798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51144
W20260619 00:53:03.980257 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3675: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:53:03.980437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3675
I20260619 00:53:03.996075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3808
Jun 19 00:53:04 dist-test-slave-r7bh krb5kdc[3435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830383, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:53:04.013031 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:03.997241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.97.3:44307)
0619 00:53:03.997923 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:03.997930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:03.997959 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:53:03.998020 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:03.998029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:03.998268 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:03.998394 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:03.999659 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:04.000727 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:04.002121 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:04.002784 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:04.002907 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:04.007445 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:04.007457 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:04.007473 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:04.007525 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:04.010338 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:04.010984 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:04.010991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:04.010993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:04.011119 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:04.011451 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:04.011454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:04.011456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:04.011789 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:04.012018 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:04.012321 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 00:53:04.012529 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":376,"threads_started":1}
W20260619 00:53:04.014323 3931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39320
W20260619 00:53:04.015412 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3808: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:53:04.015578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3808
I20260619 00:53:04.031404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3448
I20260619 00:53:04.032672 3510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:04.136174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3448
2026-06-19T00:53:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3680 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:04 dist-test-slave-r7bh krb5kdc[3953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:04 dist-test-slave-r7bh krb5kdc[3953](info): set up 2 sockets
Jun 19 00:53:04 dist-test-slave-r7bh krb5kdc[3953](info): commencing operation
krb5kdc: starting...
W20260619 00:53:05.229308 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:05 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:05.429271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39327
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:05.754020 3966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:05.754395 3966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:05.754804 3966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:05.762326 3966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:05.762493 3966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:05.762602 3966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:05.762706 3966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:05.762794 3966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:05.772898 3966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:05.774284 3966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:05.775622 3966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:05.787840 3974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:05.787774 3972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:05.788420 3971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:05.788707 3966 server_base.cc:1034] running on GCE node
I20260619 00:53:05.789707 3966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:05.791247 3966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:05.792505 3966 hybrid_clock.cc:648] HybridClock initialized: now 1781830385792437 us; error 79 us; skew 500 ppm
I20260619 00:53:05.792939 3966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:05 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:05.796458 3966 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:05.798221 3966 webserver.cc:460] Webserver started at http://127.0.97.62:36217/ using document root <none> and password file <none>
I20260619 00:53:05.798971 3966 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:05.799124 3966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:05.799535 3966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:05.802716 3966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "3d2e1c84150645da89d5e4d9dfab260b"
format_stamp: "Formatted at 2026-06-19 00:53:05 on dist-test-slave-r7bh"
server_key: "c397329f01d9b0ff573a8aaf876a3dad"
server_key_iv: "c832050f6dfb44ec9752454b45d2d8d2"
server_key_version: "encryptionkey@0"
I20260619 00:53:05.803452 3966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d2e1c84150645da89d5e4d9dfab260b"
format_stamp: "Formatted at 2026-06-19 00:53:05 on dist-test-slave-r7bh"
server_key: "c397329f01d9b0ff573a8aaf876a3dad"
server_key_iv: "c832050f6dfb44ec9752454b45d2d8d2"
server_key_version: "encryptionkey@0"
I20260619 00:53:05.808770 3966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:53:05.812346 3981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:05.813858 3966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:05.814045 3966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "3d2e1c84150645da89d5e4d9dfab260b"
format_stamp: "Formatted at 2026-06-19 00:53:05 on dist-test-slave-r7bh"
server_key: "c397329f01d9b0ff573a8aaf876a3dad"
server_key_iv: "c832050f6dfb44ec9752454b45d2d8d2"
server_key_version: "encryptionkey@0"
I20260619 00:53:05.814257 3966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:05.832643 3966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:05.837469 3966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:05.837791 3966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:05.856580 3966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39327
I20260619 00:53:05.856616 4032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39327 every 8 connection(s)
I20260619 00:53:05.858090 3966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:05.862425 4033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:05.868498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 3966
I20260619 00:53:05.868762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:05.869444 388 external_mini_cluster.cc:1428] Setting key e9bd18b52bf39ad57d10a085ad401787
I20260619 00:53:05.874543 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b: Bootstrap starting.
I20260619 00:53:05.877817 4033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:05.878994 4033 log.cc:824] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:05.882408 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b: No bootstrap required, opened a new log
Jun 19 00:53:05 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:05.892786 4033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } }
I20260619 00:53:05.893352 4033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:05.893476 4033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2e1c84150645da89d5e4d9dfab260b, State: Initialized, Role: FOLLOWER
I20260619 00:53:05.894163 4033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } }
I20260619 00:53:05.894434 4033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:05.894577 4033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:05.894752 4033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:05.894898 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:05.871648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.0.97.62:39327)
0619 00:53:05.872520 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:05.872534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:05.872587 (+ 53us) server_negotiation.cc:407] Connection header received
0619 00:53:05.874374 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:05.874398 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:05.874899 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:05.875373 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:05.876982 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:05.878497 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:05.880012 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:05.881046 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:05.881258 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:05.885960 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:05.885971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:05.885994 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:05.886048 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:05.890240 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:05.890978 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:05.890987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:05.890997 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:05.891218 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:05.891626 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:05.891636 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:05.891639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:05.892349 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:05.892620 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:05.893012 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 00:53:05.893509 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":381,"threads_started":1}
I20260619 00:53:05.897612 4033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } }
I20260619 00:53:05.898207 4033 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2e1c84150645da89d5e4d9dfab260b; no voters:
I20260619 00:53:05.898747 4033 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:05.898917 4038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:05.900336 4038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [term 1 LEADER]: Becoming Leader. State: Replica: 3d2e1c84150645da89d5e4d9dfab260b, State: Running, Role: LEADER
I20260619 00:53:05.901041 4033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:05.901010 4038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } }
I20260619 00:53:05.904093 4040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2e1c84150645da89d5e4d9dfab260b. Latest consensus state: current_term: 1 leader_uuid: "3d2e1c84150645da89d5e4d9dfab260b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } } }
I20260619 00:53:05.904505 4040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:05.905208 4039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2e1c84150645da89d5e4d9dfab260b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e1c84150645da89d5e4d9dfab260b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39327 } } }
I20260619 00:53:05.905476 4039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:05.905561 4046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:05.911960 4046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:05.922847 4046 catalog_manager.cc:1347] Generated new cluster ID: 4a7af436a3a842afa925294bb0c11c45
I20260619 00:53:05.922978 4046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:05.936278 4046 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:05.938237 4046 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:05.947672 4046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2e1c84150645da89d5e4d9dfab260b: Generated new TSK 0
I20260619 00:53:05.948612 4046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:53:06.013312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:06.346918 4059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:06.347211 4059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:06.347410 4059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:06.354619 4059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:06.354729 4059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:06.354923 4059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:06.364621 4059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:06.365990 4059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:06.367318 4059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:06.380506 4067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:06.380571 4065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:06.380523 4064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:06.381840 4059 server_base.cc:1034] running on GCE node
I20260619 00:53:06.382562 4059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:06.383630 4059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:06.384872 4059 hybrid_clock.cc:648] HybridClock initialized: now 1781830386384766 us; error 67 us; skew 500 ppm
I20260619 00:53:06.385222 4059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:06 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:06.388684 4059 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:53:06.390183 4059 webserver.cc:460] Webserver started at http://127.0.97.1:38849/ using document root <none> and password file <none>
I20260619 00:53:06.390883 4059 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:06.391014 4059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:06.391328 4059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:06.394284 4059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "c901baa46c684773a906bbf4cfc08185"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "06294679b4b53915aaf3443713fe2675"
server_key_iv: "6c47a2a2a33b6a7dba069a68701e2000"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.395092 4059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c901baa46c684773a906bbf4cfc08185"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "06294679b4b53915aaf3443713fe2675"
server_key_iv: "6c47a2a2a33b6a7dba069a68701e2000"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.400172 4059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:06.404088 4074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.405508 4059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:06.405683 4059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "c901baa46c684773a906bbf4cfc08185"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "06294679b4b53915aaf3443713fe2675"
server_key_iv: "6c47a2a2a33b6a7dba069a68701e2000"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.405866 4059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:06.422482 4059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:06.427014 4059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:06.427340 4059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:06.428346 4059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:06.429729 4059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:06.429801 4059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.429970 4059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:06.430028 4059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.462091 4059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41557
I20260619 00:53:06.462152 4186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41557 every 8 connection(s)
I20260619 00:53:06.463398 4059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:06.465631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4059
I20260619 00:53:06.465845 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:06.466670 388 external_mini_cluster.cc:1428] Setting key 2c036c539e9f133f80d96e1d39d40c5f
Jun 19 00:53:06 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:53:06.485201 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:06.466258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40951 (local address 127.0.97.62:39327)
0619 00:53:06.466745 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:06.466752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:06.467906 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 00:53:06.469805 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:06.469810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:06.469907 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:06.470024 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:06.471541 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.472242 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.474456 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.474849 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.474993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:06.479031 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:06.479036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:06.479039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:06.479129 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:06.481901 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:06.482764 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:06.482769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:06.482771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:06.482890 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:06.483359 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:06.483371 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:06.483373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:06.483642 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:06.483798 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:06.484569 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 00:53:06.484795 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260619 00:53:06.486292 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:06.466957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39327 (local address 127.0.97.1:40951)
0619 00:53:06.467746 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:06.467787 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:06.469540 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:06.470262 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:06.470272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:06.470790 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:06.471372 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:06.471389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.472414 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.472418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:06.474297 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:06.474313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.475088 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.475094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:06.475335 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:06.476219 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:06.476242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:06.478759 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:06.482094 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:06.482106 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:06.482128 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:06.482596 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:06.483013 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:06.483019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:06.483022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:06.483237 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:06.483778 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:06.483786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:06.484167 (+ 381us) client_negotiation.cc:769] Sending connection context
0619 00:53:06.484460 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 00:53:06.485072 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:53:06.488072 4187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39327
I20260619 00:53:06.488408 4187 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:06.489332 4187 heartbeater.cc:511] Master 127.0.97.62:39327 requested a full tablet report, sending...
I20260619 00:53:06.491685 3998 ts_manager.cc:194] Registered new tserver with Master: c901baa46c684773a906bbf4cfc08185 (127.0.97.1:41557)
I20260619 00:53:06.493430 3998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:53:06.513034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:06.854141 4193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:06.854429 4193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:06.854532 4193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:06.865162 4193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:06.865317 4193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:06.865502 4193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:06.880766 4193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:06.882138 4193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:06.883488 4193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:06.896945 4198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:06.897035 4199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:06.896991 4201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:06.897853 4193 server_base.cc:1034] running on GCE node
I20260619 00:53:06.898546 4193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:06.899520 4193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:06.900715 4193 hybrid_clock.cc:648] HybridClock initialized: now 1781830386900674 us; error 30 us; skew 500 ppm
I20260619 00:53:06.901147 4193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:06 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:06.904459 4193 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:53:06.906078 4193 webserver.cc:460] Webserver started at http://127.0.97.2:38409/ using document root <none> and password file <none>
I20260619 00:53:06.906803 4193 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:06.906935 4193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:06.907300 4193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:06.910880 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "e7d3349758d341bab6dca6f4845512c9"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "1d3cd521aae12115c797c4e4cb2cdfd7"
server_key_iv: "e42e1eb08d645d185309a48a3a528b23"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.911779 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d3349758d341bab6dca6f4845512c9"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "1d3cd521aae12115c797c4e4cb2cdfd7"
server_key_iv: "e42e1eb08d645d185309a48a3a528b23"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.917558 4193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:53:06.921422 4208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.922894 4193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:06.923048 4193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "e7d3349758d341bab6dca6f4845512c9"
format_stamp: "Formatted at 2026-06-19 00:53:06 on dist-test-slave-r7bh"
server_key: "1d3cd521aae12115c797c4e4cb2cdfd7"
server_key_iv: "e42e1eb08d645d185309a48a3a528b23"
server_key_version: "encryptionkey@0"
I20260619 00:53:06.923285 4193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:06.944173 4193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:06.948451 4193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:06.948800 4193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:06.949877 4193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:06.951243 4193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:06.951318 4193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.951481 4193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:06.951534 4193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:06.986663 4193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34723
I20260619 00:53:06.986779 4320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34723 every 8 connection(s)
I20260619 00:53:06.988067 4193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:06.993302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4193
I20260619 00:53:06.993527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:06.993961 388 external_mini_cluster.cc:1428] Setting key 3716ff0b80cb0b3fedbdeecee106f5fd
Jun 19 00:53:06 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:07.011708 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:06.991160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48261 (local address 127.0.97.62:39327)
0619 00:53:06.991710 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:06.991715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:06.992641 (+ 926us) server_negotiation.cc:407] Connection header received
0619 00:53:06.994822 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:06.994828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:06.994913 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:06.995045 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:06.997447 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.998188 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.999790 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.000257 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.000398 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.004923 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.004929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.004933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.004974 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.007889 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.008965 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.008973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.008981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.009166 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.009704 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.009713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.009715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.010062 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.010253 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.011092 (+ 839us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.011322 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:53:07.012885 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:06.991675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39327 (local address 127.0.97.2:48261)
0619 00:53:06.992448 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:06.992491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:06.994461 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:06.995984 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:06.995997 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:06.996578 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:06.997256 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:06.997276 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:06.998343 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:06.998348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:06.999635 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:06.999646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.000499 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.000506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:07.000748 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.001605 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:07.001634 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:07.004578 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:07.008084 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:07.008096 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:07.008117 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:07.008761 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:07.009296 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:07.009301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:07.009305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:07.009513 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:07.010256 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:07.010266 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:07.010705 (+ 439us) client_negotiation.cc:769] Sending connection context
0619 00:53:07.011008 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 00:53:07.011619 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":326,"threads_started":1}
I20260619 00:53:07.014854 4321 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:53:07.015234 4321 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:07.016084 4321 heartbeater.cc:511] Master 127.0.97.62:39327 requested a full tablet report, sending...
I20260619 00:53:07.017855 3998 ts_manager.cc:194] Registered new tserver with Master: e7d3349758d341bab6dca6f4845512c9 (127.0.97.2:34723)
I20260619 00:53:07.019152 3998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:53:07.041302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:07.365064 4327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:07.365415 4327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:07.365593 4327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:07.372938 4327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:07.373095 4327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:07.373288 4327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:07.383074 4327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:07.384407 4327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:07.385927 4327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:07.399370 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:07.399415 4335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:07.399338 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:07.400152 4327 server_base.cc:1034] running on GCE node
I20260619 00:53:07.400784 4327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:07.401988 4327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:07.403187 4327 hybrid_clock.cc:648] HybridClock initialized: now 1781830387403116 us; error 67 us; skew 500 ppm
I20260619 00:53:07.403520 4327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:07.406930 4327 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:53:07.408500 4327 webserver.cc:460] Webserver started at http://127.0.97.3:33929/ using document root <none> and password file <none>
I20260619 00:53:07.409354 4327 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:07.409457 4327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:07.409718 4327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:07.412725 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "dcfd86211e1844a7a304824b43e622cd"
format_stamp: "Formatted at 2026-06-19 00:53:07 on dist-test-slave-r7bh"
server_key: "6e2d03674a303e81fc51d351fe494b4a"
server_key_iv: "dfe1a51b6dd87add61e6a313f76784a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:07.413618 4327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dcfd86211e1844a7a304824b43e622cd"
format_stamp: "Formatted at 2026-06-19 00:53:07 on dist-test-slave-r7bh"
server_key: "6e2d03674a303e81fc51d351fe494b4a"
server_key_iv: "dfe1a51b6dd87add61e6a313f76784a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:07.419112 4327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:53:07.422820 4342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:07.424288 4327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:07.424510 4327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "dcfd86211e1844a7a304824b43e622cd"
format_stamp: "Formatted at 2026-06-19 00:53:07 on dist-test-slave-r7bh"
server_key: "6e2d03674a303e81fc51d351fe494b4a"
server_key_iv: "dfe1a51b6dd87add61e6a313f76784a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:07.424747 4327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:07.444969 4327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:07.449208 4327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:07.449534 4327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:07.450469 4327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:07.451783 4327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:07.451857 4327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:07.451972 4327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:07.452018 4327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:07.484258 4327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44063
I20260619 00:53:07.484305 4454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44063 every 8 connection(s)
I20260619 00:53:07.485666 4327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:07.490537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4327
I20260619 00:53:07.490736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:07.491284 388 external_mini_cluster.cc:1428] Setting key 4407294d601a14abd67bf97bd4636160
I20260619 00:53:07.496529 4187 heartbeater.cc:503] Master 127.0.97.62:39327 was elected leader, sending a full tablet report...
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:07.506675 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.488304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41077 (local address 127.0.97.62:39327)
0619 00:53:07.488471 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.488476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.489779 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 00:53:07.492244 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.492248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.492345 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.492454 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.494339 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.495051 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.496583 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.497136 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.497258 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.500762 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.500767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.500771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.500848 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.503372 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.504053 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.504057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.504059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.504173 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.504557 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.504560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.504562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.504888 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.505059 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.505985 (+ 926us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.506253 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 00:53:07.507627 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.488748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39327 (local address 127.0.97.3:41077)
0619 00:53:07.489547 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:07.489592 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:07.491939 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:07.492747 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:07.492760 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:07.493536 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:07.494171 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:07.494196 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.495201 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.495206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:07.496443 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:07.496455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.497473 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.497477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:07.497674 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.498439 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:07.498467 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:07.500546 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:07.503511 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:07.503518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:07.503531 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:07.503942 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:07.504268 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:07.504271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:07.504273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:07.504443 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:07.505041 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:07.505051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:07.505507 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 00:53:07.505813 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 00:53:07.506374 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":288,"threads_started":1}
I20260619 00:53:07.508983 4455 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39327
I20260619 00:53:07.509296 4455 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:07.510033 4455 heartbeater.cc:511] Master 127.0.97.62:39327 requested a full tablet report, sending...
I20260619 00:53:07.511430 3998 ts_manager.cc:194] Registered new tserver with Master: dcfd86211e1844a7a304824b43e622cd (127.0.97.3:44063)
I20260619 00:53:07.512233 3998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41077
I20260619 00:53:07.519433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:07.569860 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.554667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.97.1:41557)
0619 00:53:07.555301 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.555312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.555347 (+ 35us) server_negotiation.cc:407] Connection header received
0619 00:53:07.555426 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.555433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.555651 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.555823 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.557283 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.558300 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.559533 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.560114 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.560257 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.564188 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.564199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.564214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.564254 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.567099 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.567735 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.567743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.567745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.567867 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.568248 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.568255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.568258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.568589 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.568931 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.569235 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.569504 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:07.588181 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.574186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.0.97.62:39327)
0619 00:53:07.574400 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.574404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.574522 (+ 118us) server_negotiation.cc:407] Connection header received
0619 00:53:07.574613 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.574617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.574679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.574764 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.576024 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.576740 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.578179 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.578574 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.578681 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.582614 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.582619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.582622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.582671 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.585571 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.586249 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.586253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.586256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.586389 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.586760 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.586763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.586765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.587112 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.587296 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.587590 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.587779 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 00:53:07.588797 3998 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51172
I20260619 00:53:07.589927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4059
I20260619 00:53:07.602738 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.590822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.0.97.1:41557)
0619 00:53:07.590984 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.590988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.591187 (+ 199us) server_negotiation.cc:407] Connection header received
0619 00:53:07.591325 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.591331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.591414 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.591534 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.592782 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.593725 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.594880 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.595289 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.595383 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.597595 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.597600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.597603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.597633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.600188 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.600893 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.600902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.600904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.601024 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.601397 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.601410 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.601412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.601673 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.601862 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.602250 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.602480 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 00:53:07.603358 4182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:07.685639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4059
I20260619 00:53:07.701452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4193
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:53:07.717906 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.702465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.0.97.2:34723)
0619 00:53:07.703023 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.703030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.703055 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:53:07.703126 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.703132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.703342 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.703524 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.704696 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.705791 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.707198 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.707771 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.707910 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.712044 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.712055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.712070 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.712114 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.715034 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.715679 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.715686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.715688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.715804 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.716166 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.716170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.716173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.716564 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.716790 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.717207 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.717504 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
I20260619 00:53:07.719266 4316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:07.801394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4193
I20260619 00:53:07.817238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4327
Jun 19 00:53:07 dist-test-slave-r7bh krb5kdc[3953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:53:07.837867 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:07.818416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.0.97.3:44063)
0619 00:53:07.819048 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:07.819055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:07.819086 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:53:07.819146 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:07.819152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:07.819356 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:07.819490 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:07.820686 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.821753 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.823038 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:07.823580 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:07.823674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:07.831632 (+ 7958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:07.831643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:07.831659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:07.831701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:07.834755 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.835516 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.835524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.835526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.835676 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:07.836127 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:07.836132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:07.836134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:07.836506 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:07.836769 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:07.837220 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 00:53:07.837438 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":324,"threads_started":1}
I20260619 00:53:07.839283 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:07.923940 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4327
I20260619 00:53:07.940114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 3966
I20260619 00:53:07.941499 4028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:08.045890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 3966
2026-06-19T00:53:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3910 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:08 dist-test-slave-r7bh krb5kdc[4474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:08 dist-test-slave-r7bh krb5kdc[4474](info): set up 2 sockets
Jun 19 00:53:08 dist-test-slave-r7bh krb5kdc[4474](info): commencing operation
krb5kdc: starting...
W20260619 00:53:09.149405 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:09 dist-test-slave-r7bh krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:09.360351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41937
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:09.708909 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:09.709226 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:09.709388 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:09.716854 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:09.717003 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:09.717113 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:09.717208 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:09.717278 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:09.727913 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:09.729419 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:09.730767 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:09.742476 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:09.742708 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:09.742739 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:09.743611 4487 server_base.cc:1034] running on GCE node
I20260619 00:53:09.744453 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:09.745993 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:09.747251 4487 hybrid_clock.cc:648] HybridClock initialized: now 1781830389747183 us; error 69 us; skew 500 ppm
I20260619 00:53:09.747608 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:09 dist-test-slave-r7bh krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:09.750803 4487 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:09.752085 4487 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:09.752183 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:09.752440 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:09.755471 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "8e119068116d480ba6a57698b339e2c6"
format_stamp: "Formatted at 2026-06-19 00:53:09 on dist-test-slave-r7bh"
server_key: "08954857a8cff2b28d06dea823f9892e"
server_key_iv: "f6e2856db9c8eb5e8514d592dc0220fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:09.756214 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "8e119068116d480ba6a57698b339e2c6"
format_stamp: "Formatted at 2026-06-19 00:53:09 on dist-test-slave-r7bh"
server_key: "08954857a8cff2b28d06dea823f9892e"
server_key_iv: "f6e2856db9c8eb5e8514d592dc0220fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:09.761279 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:53:09.764803 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:09.766304 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:09.766450 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "8e119068116d480ba6a57698b339e2c6"
format_stamp: "Formatted at 2026-06-19 00:53:09 on dist-test-slave-r7bh"
server_key: "08954857a8cff2b28d06dea823f9892e"
server_key_iv: "f6e2856db9c8eb5e8514d592dc0220fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:09.766636 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:09.783358 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:09.787689 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:09.788002 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:09.807108 4487 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41937
I20260619 00:53:09.807128 4552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41937 every 8 connection(s)
I20260619 00:53:09.808084 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:09.810613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4487
I20260619 00:53:09.810804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:09.811470 388 external_mini_cluster.cc:1428] Setting key 22bf627d82e5d898a72cf48209d3a304
I20260619 00:53:09.812533 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 00:53:09 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:09.826553 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: Bootstrap starting.
I20260619 00:53:09.830735 4553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:09.832010 4553 log.cc:824] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:09.835470 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: No bootstrap required, opened a new log
I20260619 00:53:09.835857 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:09.813526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.97.62:41937)
0619 00:53:09.814318 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:09.814333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:09.814381 (+ 48us) server_negotiation.cc:407] Connection header received
0619 00:53:09.816128 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:09.816156 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:09.816710 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:09.817217 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:09.818599 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:09.820649 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:09.822048 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:09.823167 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:09.823351 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:09.827280 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:09.827291 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:09.827311 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:09.827356 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:09.831362 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:09.832090 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:09.832101 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:09.832111 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:09.832313 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:09.832775 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:09.832780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:09.832783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:09.833448 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:09.833731 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:09.834130 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 00:53:09.834552 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":328,"threads_started":1}
I20260619 00:53:09.843721 4553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } }
I20260619 00:53:09.844158 4553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:09.844336 4553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e119068116d480ba6a57698b339e2c6, State: Initialized, Role: FOLLOWER
I20260619 00:53:09.844973 4553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } }
I20260619 00:53:09.845186 4553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:09.845289 4553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:09.845413 4553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:09.847470 4553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } }
I20260619 00:53:09.847986 4553 leader_election.cc:302] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e119068116d480ba6a57698b339e2c6; no voters:
I20260619 00:53:09.848570 4553 leader_election.cc:288] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:09.848721 4558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:09.850833 4553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:09.852285 4558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8e119068116d480ba6a57698b339e2c6, State: Running, Role: LEADER
I20260619 00:53:09.853117 4558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } }
I20260619 00:53:09.856091 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e119068116d480ba6a57698b339e2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } } }
I20260619 00:53:09.856067 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e119068116d480ba6a57698b339e2c6. Latest consensus state: current_term: 1 leader_uuid: "8e119068116d480ba6a57698b339e2c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e119068116d480ba6a57698b339e2c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41937 } } }
I20260619 00:53:09.857005 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:09.857002 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6 [sys.catalog]: This master's current role is: LEADER
W20260619 00:53:09.861174 4572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:53:09.861271 4572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:53:09.861421 4573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:09.863019 4573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:09.871795 4573 catalog_manager.cc:1347] Generated new cluster ID: 4287fc695423472cb69ec2f48028d50a
I20260619 00:53:09.872042 4573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:09.895668 4573 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:09.897672 4573 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:09.908452 4573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: Generated new TSK 0
I20260619 00:53:09.909533 4573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:53:09.965062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:10.297605 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:10.297891 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:10.298127 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:10.305495 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:10.305649 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:10.305850 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:10.315734 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:10.317098 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:10.318449 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:10.331683 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:10.331718 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:10.331647 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:10.332585 4579 server_base.cc:1034] running on GCE node
I20260619 00:53:10.333376 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:10.334420 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:10.335731 4579 hybrid_clock.cc:648] HybridClock initialized: now 1781830390335658 us; error 62 us; skew 500 ppm
I20260619 00:53:10.336051 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:10 dist-test-slave-r7bh krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:10.339354 4579 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:53:10.340636 4579 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:10.340752 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:10.341128 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:10.344112 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "1e41733e742944f785700bff67046e9e"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "311da96e64427eaf75760279b62a275e"
server_key_iv: "6c9c16b709ac3bf5d73d5dd2ccf9aa9e"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.345033 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1e41733e742944f785700bff67046e9e"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "311da96e64427eaf75760279b62a275e"
server_key_iv: "6c9c16b709ac3bf5d73d5dd2ccf9aa9e"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.350080 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 00:53:10.353902 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.355408 4579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:10.355605 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "1e41733e742944f785700bff67046e9e"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "311da96e64427eaf75760279b62a275e"
server_key_iv: "6c9c16b709ac3bf5d73d5dd2ccf9aa9e"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.355794 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:10.370496 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:10.374852 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:10.375164 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:10.376106 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:10.377441 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:10.377526 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.377683 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:10.377779 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.408727 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34243
I20260619 00:53:10.408859 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34243 every 8 connection(s)
I20260619 00:53:10.409633 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:10.416260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4579
I20260619 00:53:10.416476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:10.417162 388 external_mini_cluster.cc:1428] Setting key 1b3783444e6854855f5c28539c000d74
Jun 19 00:53:10 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:10.431581 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:10.412399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33377 (local address 127.0.97.62:41937)
0619 00:53:10.412879 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:10.412885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:10.413937 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 00:53:10.415667 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:10.415672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:10.415736 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:10.415884 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:10.417518 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.418401 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.420263 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.421078 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.421224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:10.424600 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:10.424609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:10.424614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:10.424660 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:10.428307 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:10.429064 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:10.429073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:10.429079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:10.429228 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:10.429617 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:10.429625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:10.429630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:10.430009 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:10.430163 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:10.430836 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 00:53:10.431088 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260619 00:53:10.432245 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:10.413094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41937 (local address 127.0.97.1:33377)
0619 00:53:10.413769 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:10.413806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:10.415378 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:10.416064 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:10.416072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:10.416685 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:10.417344 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:10.417363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.418588 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.418593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:10.420113 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:10.420125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.421302 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.421307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:10.421507 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:10.422196 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:10.422223 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:10.424355 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:10.428455 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:10.428462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:10.428475 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:10.428950 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:10.429335 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:10.429339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:10.429341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:10.429503 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:10.430131 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:10.430136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:10.430443 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 00:53:10.430671 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 00:53:10.431047 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":280,"threads_started":1}
I20260619 00:53:10.433925 4706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41937
I20260619 00:53:10.434223 4706 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:10.434851 4706 heartbeater.cc:511] Master 127.0.97.62:41937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:53:10.437213 4518 ts_manager.cc:194] Registered new tserver with Master: 1e41733e742944f785700bff67046e9e (127.0.97.1:34243)
I20260619 00:53:10.438735 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:53:10.465029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:10.794838 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:10.795221 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:10.795454 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:10.803298 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:10.803457 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:10.803673 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:10.814432 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:10.815794 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:10.817255 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:10.830513 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:10.830485 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:10.830574 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:10.831307 4712 server_base.cc:1034] running on GCE node
I20260619 00:53:10.831961 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:10.832990 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:10.834251 4712 hybrid_clock.cc:648] HybridClock initialized: now 1781830390834199 us; error 44 us; skew 500 ppm
I20260619 00:53:10.834622 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:10 dist-test-slave-r7bh krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:10.837800 4712 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:53:10.838937 4712 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:10.839080 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:10.839392 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:10.842319 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "baeca73209a547b79a4edcfa6499d524"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "61b6b7a688f945ddec45cafd0709f885"
server_key_iv: "d08c01fa46962df0fbf8f71cdac103c6"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.843178 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "baeca73209a547b79a4edcfa6499d524"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "61b6b7a688f945ddec45cafd0709f885"
server_key_iv: "d08c01fa46962df0fbf8f71cdac103c6"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.848266 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 00:53:10.852002 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.853557 4712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:10.853737 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "baeca73209a547b79a4edcfa6499d524"
format_stamp: "Formatted at 2026-06-19 00:53:10 on dist-test-slave-r7bh"
server_key: "61b6b7a688f945ddec45cafd0709f885"
server_key_iv: "d08c01fa46962df0fbf8f71cdac103c6"
server_key_version: "encryptionkey@0"
I20260619 00:53:10.853952 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:10.870106 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:10.874547 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:10.874856 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:10.875808 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:10.877175 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:10.877259 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.877409 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:10.877506 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:10.907866 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36557
I20260619 00:53:10.907908 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36557 every 8 connection(s)
I20260619 00:53:10.908727 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:10.913803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4712
I20260619 00:53:10.913995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:10.914449 388 external_mini_cluster.cc:1428] Setting key 4b9c9d8ca2d36ff7c66fe0d72d23d2af
Jun 19 00:53:10 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:10.931555 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:10.911509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47751 (local address 127.0.97.62:41937)
0619 00:53:10.911724 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:10.911731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:10.913104 (+ 1373us) server_negotiation.cc:407] Connection header received
0619 00:53:10.915129 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:10.915133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:10.915198 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:10.915343 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:10.917046 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.917748 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.920247 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.920674 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.920776 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:10.924461 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:10.924470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:10.924478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:10.924522 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:10.928025 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:10.928735 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:10.928744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:10.928750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:10.928927 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:10.929324 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:10.929332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:10.929338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:10.929720 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:10.929872 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:10.930860 (+ 988us) server_negotiation.cc:299] Negotiation successful
0619 00:53:10.931116 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:53:10.932335 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:10.912066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41937 (local address 127.0.97.2:47751)
0619 00:53:10.912907 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:10.912965 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:10.914798 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:10.915573 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:10.915582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:10.916095 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:10.916847 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:10.916876 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.917916 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.917925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:10.920080 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:10.920097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:10.920932 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:10.920941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:10.921261 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:10.921991 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:10.922029 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:10.924237 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:10.928169 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:10.928176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:10.928189 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:10.928592 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:10.929032 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:10.929035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:10.929037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:10.929202 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:10.929839 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:10.929844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:10.930158 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 00:53:10.930703 (+ 545us) client_negotiation.cc:240] Negotiation successful
0619 00:53:10.931139 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260619 00:53:10.933914 4839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41937
I20260619 00:53:10.934257 4839 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:10.935111 4839 heartbeater.cc:511] Master 127.0.97.62:41937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:53:10.936707 4518 ts_manager.cc:194] Registered new tserver with Master: baeca73209a547b79a4edcfa6499d524 (127.0.97.2:36557)
I20260619 00:53:10.937636 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:53:10.960429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:11.297983 4844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:11.298344 4844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:11.298543 4844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:11.306195 4844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:11.306373 4844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:11.306579 4844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:11.316661 4844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:11.318099 4844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:11.319479 4844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:11.333935 4852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:11.334081 4849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:11.334057 4850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:11.334900 4844 server_base.cc:1034] running on GCE node
I20260619 00:53:11.335489 4844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:11.336422 4844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:11.337649 4844 hybrid_clock.cc:648] HybridClock initialized: now 1781830391337584 us; error 63 us; skew 500 ppm
I20260619 00:53:11.337970 4844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:11 dist-test-slave-r7bh krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:11.341130 4844 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:53:11.342356 4844 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:11.342492 4844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:11.342825 4844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:11.345788 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "920a1990d233424780d08de142e25778"
format_stamp: "Formatted at 2026-06-19 00:53:11 on dist-test-slave-r7bh"
server_key: "bf3497eaa5921d901689d21f004a1cb5"
server_key_iv: "ce32cc2bc7947b3bf8bc64b1c5fc6c48"
server_key_version: "encryptionkey@0"
I20260619 00:53:11.346606 4844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "920a1990d233424780d08de142e25778"
format_stamp: "Formatted at 2026-06-19 00:53:11 on dist-test-slave-r7bh"
server_key: "bf3497eaa5921d901689d21f004a1cb5"
server_key_iv: "ce32cc2bc7947b3bf8bc64b1c5fc6c48"
server_key_version: "encryptionkey@0"
I20260619 00:53:11.351811 4844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:11.355428 4858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:11.356894 4844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:11.357120 4844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "920a1990d233424780d08de142e25778"
format_stamp: "Formatted at 2026-06-19 00:53:11 on dist-test-slave-r7bh"
server_key: "bf3497eaa5921d901689d21f004a1cb5"
server_key_iv: "ce32cc2bc7947b3bf8bc64b1c5fc6c48"
server_key_version: "encryptionkey@0"
I20260619 00:53:11.357360 4844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:11.382628 4844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:11.387339 4844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:11.387665 4844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:11.388768 4844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:11.390314 4844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:11.390451 4844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:11.390606 4844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:11.390705 4844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:11.426520 4844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45741
I20260619 00:53:11.426568 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45741 every 8 connection(s)
I20260619 00:53:11.427431 4844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:11.434926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 4844
I20260619 00:53:11.435158 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:11.435570 388 external_mini_cluster.cc:1428] Setting key 951ebdc08fb837ba3ca3f8352a60369f
Jun 19 00:53:11 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:11.442279 4706 heartbeater.cc:503] Master 127.0.97.62:41937 was elected leader, sending a full tablet report...
I20260619 00:53:11.448681 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.430045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36109 (local address 127.0.97.62:41937)
0619 00:53:11.430443 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.430448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.431461 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 00:53:11.433353 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.433358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.433452 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.433564 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:11.435151 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.436014 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.438404 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.438946 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.439088 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.442758 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:11.442766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:11.442770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:11.442812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:11.445805 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.446588 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.446592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.446594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.446714 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.447092 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.447095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.447103 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.447366 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:11.447476 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.448263 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.448440 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 00:53:11.450275 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.430512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41937 (local address 127.0.97.3:36109)
0619 00:53:11.431264 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.431304 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.433069 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.433804 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.433813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.434337 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:11.434907 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.434924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.436182 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.436186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.438260 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.438273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.439183 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.439190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.439426 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.440088 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:11.440113 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:11.442485 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:11.445955 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:11.445963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:11.445978 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:11.446472 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:11.446804 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:11.446806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:11.446809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:11.446990 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:11.447526 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:11.447532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:11.447890 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.448138 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.448566 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":321,"threads_started":1}
I20260619 00:53:11.451843 4971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41937
I20260619 00:53:11.452117 4971 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:11.452801 4971 heartbeater.cc:511] Master 127.0.97.62:41937 requested a full tablet report, sending...
I20260619 00:53:11.454391 4518 ts_manager.cc:194] Registered new tserver with Master: 920a1990d233424780d08de142e25778 (127.0.97.3:45741)
I20260619 00:53:11.455210 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36109
I20260619 00:53:11.463802 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:53:11.481196 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.468495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.0.97.62:41937)
0619 00:53:11.468708 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.468716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.469267 (+ 551us) server_negotiation.cc:407] Connection header received
0619 00:53:11.469486 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.469490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.469552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.469657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:11.471132 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.471832 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.473122 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.473569 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.473699 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.476072 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:11.476078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:11.476081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:11.476113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:11.478536 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.479199 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.479208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.479211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.479333 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.479704 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.479709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.479712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.480013 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:11.480282 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.480626 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.480894 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 00:53:11.485550 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:11.492274 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:11.528247 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.505368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.97.3:45741)
0619 00:53:11.516720 (+ 11352us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.516731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.517414 (+ 683us) server_negotiation.cc:407] Connection header received
0619 00:53:11.517520 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.517530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.517818 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.518050 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:11.519324 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.521472 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.524833 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.526271 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.526423 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.526694 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.527697 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.527899 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11139,"thread_start_us":10981,"threads_started":1}
I20260619 00:53:11.528239 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.504962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45741 (local address 127.0.0.1:56886)
0619 00:53:11.516960 (+ 11998us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.517017 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.517208 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.518457 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.518474 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.518516 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:11.519026 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.519042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.521682 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.521691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.524645 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.524666 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.526480 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.526486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.527427 (+ 941us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.527480 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.527645 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.527848 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11614,"thread_start_us":11779,"threads_started":1}
I20260619 00:53:11.530400 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.505334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34243 (local address 127.0.0.1:48238)
0619 00:53:11.517053 (+ 11719us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.517072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.517312 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.519864 (+ 2552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.519872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.519910 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:11.520479 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.520495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.525001 (+ 4506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.525006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.527062 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.527076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.529237 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.529457 (+ 220us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.529661 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.529698 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.529793 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.529941 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11594,"mutex_wait_us":1,"thread_start_us":11427,"threads_started":1}
I20260619 00:53:11.531167 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.506331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.0.97.1:34243)
0619 00:53:11.516337 (+ 10006us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.516350 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.517194 (+ 844us) server_negotiation.cc:407] Connection header received
0619 00:53:11.517509 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.517516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.519227 (+ 1711us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.519451 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:11.520757 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.524753 (+ 3996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.527249 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.529017 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.529162 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.529377 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.530724 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.530927 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9783,"thread_start_us":9598,"threads_started":1}
I20260619 00:53:11.533290 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.517694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36557 (local address 127.0.0.1:45408)
0619 00:53:11.524270 (+ 6576us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.524289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.524390 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.525877 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.525883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.525909 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:11.526297 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.526309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.528382 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.528388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.530723 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.530738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.532442 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.532449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.532649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.532685 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.532789 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.532959 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6429,"thread_start_us":6765,"threads_started":1}
I20260619 00:53:11.533360 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.517985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.0.97.2:36557)
0619 00:53:11.525221 (+ 7236us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.525232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.525261 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:53:11.525361 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.525369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.525640 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.525856 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:11.526506 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.528177 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.530888 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.532228 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.532358 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.532526 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.532870 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.533032 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7081,"thread_start_us":6957,"threads_started":1}
I20260619 00:53:11.532994 4906 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc06e3878478ebfd34d42b268dabe (DEFAULT_TABLE table=test-table [id=455b257869324ba6a5bc905eb5457c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:11.534498 4906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2bc06e3878478ebfd34d42b268dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:11.534874 4641 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc06e3878478ebfd34d42b268dabe (DEFAULT_TABLE table=test-table [id=455b257869324ba6a5bc905eb5457c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:11.536396 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2bc06e3878478ebfd34d42b268dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:11.536906 4774 tablet_service.cc:1467] Processing CreateTablet for tablet fc2bc06e3878478ebfd34d42b268dabe (DEFAULT_TABLE table=test-table [id=455b257869324ba6a5bc905eb5457c4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:11.538290 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2bc06e3878478ebfd34d42b268dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:11.547757 4989 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Bootstrap starting.
I20260619 00:53:11.551213 4989 tablet_bootstrap.cc:654] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:11.552561 4989 log.cc:824] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:11.552919 4990 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Bootstrap starting.
I20260619 00:53:11.555014 4992 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Bootstrap starting.
I20260619 00:53:11.557126 4990 tablet_bootstrap.cc:654] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:11.557235 4989 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: No bootstrap required, opened a new log
I20260619 00:53:11.557528 4989 ts_tablet_manager.cc:1397] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260619 00:53:11.558493 4990 log.cc:824] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:11.560070 4992 tablet_bootstrap.cc:654] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:11.561496 4992 log.cc:824] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:11.562022 4990 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: No bootstrap required, opened a new log
I20260619 00:53:11.562351 4990 ts_tablet_manager.cc:1397] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 00:53:11.564677 4992 tablet_bootstrap.cc:492] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: No bootstrap required, opened a new log
I20260619 00:53:11.564989 4992 ts_tablet_manager.cc:1397] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 00:53:11.567198 4989 raft_consensus.cc:348] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.567598 4989 raft_consensus.cc:374] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:11.567705 4989 raft_consensus.cc:729] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920a1990d233424780d08de142e25778, State: Initialized, Role: FOLLOWER
I20260619 00:53:11.568389 4989 consensus_queue.cc:260] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.569590 4971 heartbeater.cc:503] Master 127.0.97.62:41937 was elected leader, sending a full tablet report...
I20260619 00:53:11.570469 4989 ts_tablet_manager.cc:1428] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260619 00:53:11.574523 4990 raft_consensus.cc:348] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.574947 4990 raft_consensus.cc:374] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:11.575058 4990 raft_consensus.cc:729] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e41733e742944f785700bff67046e9e, State: Initialized, Role: FOLLOWER
I20260619 00:53:11.575735 4990 consensus_queue.cc:260] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.577189 4992 raft_consensus.cc:348] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.577513 4990 ts_tablet_manager.cc:1428] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 00:53:11.577652 4992 raft_consensus.cc:374] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:11.577766 4992 raft_consensus.cc:729] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeca73209a547b79a4edcfa6499d524, State: Initialized, Role: FOLLOWER
I20260619 00:53:11.578410 4992 consensus_queue.cc:260] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.579442 4839 heartbeater.cc:503] Master 127.0.97.62:41937 was elected leader, sending a full tablet report...
I20260619 00:53:11.580309 4992 ts_tablet_manager.cc:1428] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260619 00:53:11.660763 4840 tablet.cc:2369] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:11.662154 4707 tablet.cc:2369] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:11.678998 4972 tablet.cc:2369] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:11.807451 4997 raft_consensus.cc:482] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:11.807782 4997 raft_consensus.cc:504] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.809717 4997 leader_election.cc:288] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920a1990d233424780d08de142e25778 (127.0.97.3:45741), 1e41733e742944f785700bff67046e9e (127.0.97.1:34243)
I20260619 00:53:11.817610 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.810225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42317 (local address 127.0.97.1:34243)
0619 00:53:11.810480 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.810487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.811405 (+ 918us) server_negotiation.cc:407] Connection header received
0619 00:53:11.811577 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.811584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.811667 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.811747 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:11.812467 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.813589 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.815866 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.816590 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.816748 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.816885 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.817212 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.817347 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 00:53:11.817895 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.810270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57847 (local address 127.0.97.3:45741)
0619 00:53:11.810632 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.810637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.810992 (+ 355us) server_negotiation.cc:407] Connection header received
0619 00:53:11.811356 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.811382 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.811483 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.811610 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:11.812218 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.813231 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.815737 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.817022 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.817144 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.817241 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.817531 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.817655 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260619 00:53:11.817997 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.809949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45741 (local address 127.0.97.2:57847)
0619 00:53:11.810867 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.810897 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.811140 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.811645 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.811651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.811682 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:11.812059 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.812070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.813362 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.813368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.815569 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.815583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.817227 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.817232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.817380 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.817408 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.817497 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.817634 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":26,"thread_start_us":685,"threads_started":1}
I20260619 00:53:11.818696 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.810236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34243 (local address 127.0.97.2:42317)
0619 00:53:11.811294 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:11.811315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:11.811442 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:11.811809 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:11.811814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:11.811842 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:11.812293 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.812305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.813718 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.813723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.815729 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:11.815740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.816805 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.816835 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:11.817009 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.817039 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:53:11.818312 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0619 00:53:11.818440 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":713,"threads_started":1}
I20260619 00:53:11.818975 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc06e3878478ebfd34d42b268dabe" candidate_uuid: "baeca73209a547b79a4edcfa6499d524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920a1990d233424780d08de142e25778" is_pre_election: true
I20260619 00:53:11.819505 4926 raft_consensus.cc:2454] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baeca73209a547b79a4edcfa6499d524 in term 0.
I20260619 00:53:11.819599 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc06e3878478ebfd34d42b268dabe" candidate_uuid: "baeca73209a547b79a4edcfa6499d524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e41733e742944f785700bff67046e9e" is_pre_election: true
I20260619 00:53:11.820173 4661 raft_consensus.cc:2454] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baeca73209a547b79a4edcfa6499d524 in term 0.
I20260619 00:53:11.820222 4728 leader_election.cc:302] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920a1990d233424780d08de142e25778, baeca73209a547b79a4edcfa6499d524; no voters:
I20260619 00:53:11.820696 4997 raft_consensus.cc:2781] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:11.820880 4997 raft_consensus.cc:482] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:11.820986 4997 raft_consensus.cc:3037] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:11.823127 4997 raft_consensus.cc:504] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.823784 4997 leader_election.cc:288] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [CANDIDATE]: Term 1 election: Requested vote from peers 920a1990d233424780d08de142e25778 (127.0.97.3:45741), 1e41733e742944f785700bff67046e9e (127.0.97.1:34243)
I20260619 00:53:11.824422 4926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc06e3878478ebfd34d42b268dabe" candidate_uuid: "baeca73209a547b79a4edcfa6499d524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920a1990d233424780d08de142e25778"
I20260619 00:53:11.824421 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2bc06e3878478ebfd34d42b268dabe" candidate_uuid: "baeca73209a547b79a4edcfa6499d524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e41733e742944f785700bff67046e9e"
I20260619 00:53:11.824654 4661 raft_consensus.cc:3037] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:11.824651 4926 raft_consensus.cc:3037] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:11.827899 4926 raft_consensus.cc:2454] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baeca73209a547b79a4edcfa6499d524 in term 1.
I20260619 00:53:11.827898 4661 raft_consensus.cc:2454] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baeca73209a547b79a4edcfa6499d524 in term 1.
I20260619 00:53:11.828426 4728 leader_election.cc:302] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920a1990d233424780d08de142e25778, baeca73209a547b79a4edcfa6499d524; no voters:
I20260619 00:53:11.828853 4997 raft_consensus.cc:2781] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:11.829336 4997 raft_consensus.cc:686] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 1 LEADER]: Becoming Leader. State: Replica: baeca73209a547b79a4edcfa6499d524, State: Running, Role: LEADER
I20260619 00:53:11.829820 4997 consensus_queue.cc:237] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } }
I20260619 00:53:11.834095 4517 catalog_manager.cc:5495] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 reported cstate change: term changed from 0 to 1, leader changed from <none> to baeca73209a547b79a4edcfa6499d524 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "baeca73209a547b79a4edcfa6499d524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baeca73209a547b79a4edcfa6499d524" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36557 } health_report { overall_health: HEALTHY } } }
I20260619 00:53:11.885794 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.880437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.0.97.2:36557)
0619 00:53:11.880660 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.880669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.880751 (+ 82us) server_negotiation.cc:407] Connection header received
0619 00:53:11.880966 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.880970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.881047 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.881183 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:11.881502 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.882208 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.883543 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.883997 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.884116 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.884616 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:11.885192 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:11.885328 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.885388 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.885461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:53:11.894356 4661 raft_consensus.cc:1261] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 1 FOLLOWER]: Refusing update from remote peer baeca73209a547b79a4edcfa6499d524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:11.894363 4926 raft_consensus.cc:1261] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 1 FOLLOWER]: Refusing update from remote peer baeca73209a547b79a4edcfa6499d524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:11.895315 5000 consensus_queue.cc:1035] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920a1990d233424780d08de142e25778" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:11.895646 4997 consensus_queue.cc:1035] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e41733e742944f785700bff67046e9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:11.926915 4518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:11.927255 4518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:11.931493 4518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e119068116d480ba6a57698b339e2c6: Sending DeleteTablet for 3 replicas of tablet fc2bc06e3878478ebfd34d42b268dabe
I20260619 00:53:11.932742 4906 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2bc06e3878478ebfd34d42b268dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56886
I20260619 00:53:11.932838 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2bc06e3878478ebfd34d42b268dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48238
I20260619 00:53:11.934122 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2bc06e3878478ebfd34d42b268dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45408
I20260619 00:53:11.940893 5014 tablet_replica.cc:330] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: stopping tablet replica
I20260619 00:53:11.941385 5014 raft_consensus.cc:2229] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:11.942229 5014 raft_consensus.cc:2258] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:11.942931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4579
I20260619 00:53:11.943096 5012 tablet_replica.cc:330] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: stopping tablet replica
I20260619 00:53:11.943563 5012 raft_consensus.cc:2229] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:11.944123 5012 raft_consensus.cc:2258] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:11.944993 5013 tablet_replica.cc:330] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: stopping tablet replica
I20260619 00:53:11.945451 5013 raft_consensus.cc:2229] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:11.946004 5013 raft_consensus.cc:2258] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:11.949177 5012 ts_tablet_manager.cc:1905] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:11.949184 5014 ts_tablet_manager.cc:1905] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:11.954807 5013 ts_tablet_manager.cc:1905] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 00:53:11 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:11.957593 5012 ts_tablet_manager.cc:1918] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:11.957827 5012 log.cc:1196] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/fc2bc06e3878478ebfd34d42b268dabe
I20260619 00:53:11.958464 5012 ts_tablet_manager.cc:1939] T fc2bc06e3878478ebfd34d42b268dabe P 1e41733e742944f785700bff67046e9e: Deleting consensus metadata
I20260619 00:53:11.961637 4505 catalog_manager.cc:4841] TS 1e41733e742944f785700bff67046e9e (127.0.97.1:34243): tablet fc2bc06e3878478ebfd34d42b268dabe (table test-table [id=455b257869324ba6a5bc905eb5457c4c]) successfully deleted
I20260619 00:53:11.964762 5013 ts_tablet_manager.cc:1918] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:11.964890 5014 ts_tablet_manager.cc:1918] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:11.965049 5013 log.cc:1196] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/fc2bc06e3878478ebfd34d42b268dabe
I20260619 00:53:11.965081 5014 log.cc:1196] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/fc2bc06e3878478ebfd34d42b268dabe
I20260619 00:53:11.965673 5013 ts_tablet_manager.cc:1939] T fc2bc06e3878478ebfd34d42b268dabe P 920a1990d233424780d08de142e25778: Deleting consensus metadata
I20260619 00:53:11.965679 5014 ts_tablet_manager.cc:1939] T fc2bc06e3878478ebfd34d42b268dabe P baeca73209a547b79a4edcfa6499d524: Deleting consensus metadata
I20260619 00:53:11.966732 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:11.945306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.97.1:34243)
0619 00:53:11.945521 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:11.945528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:11.945553 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:53:11.945626 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:11.945631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:11.945717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:11.945839 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:11.947859 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.949038 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.950971 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:11.951716 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:11.951823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:11.957660 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:11.957679 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:11.957697 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:11.957755 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:11.962406 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.963508 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.963519 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.963522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.963705 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:11.964212 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:11.964217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:11.964221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:11.964694 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:11.965303 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:11.966051 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 00:53:11.966305 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:53:11.967692 4503 catalog_manager.cc:4841] TS baeca73209a547b79a4edcfa6499d524 (127.0.97.2:36557): tablet fc2bc06e3878478ebfd34d42b268dabe (table test-table [id=455b257869324ba6a5bc905eb5457c4c]) successfully deleted
I20260619 00:53:11.968227 4503 catalog_manager.cc:4841] TS 920a1990d233424780d08de142e25778 (127.0.97.3:45741): tablet fc2bc06e3878478ebfd34d42b268dabe (table test-table [id=455b257869324ba6a5bc905eb5457c4c]) successfully deleted
I20260619 00:53:11.968545 4701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:12.059139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4579
I20260619 00:53:12.077169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4712
Jun 19 00:53:12 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:53:12.095108 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:12.078043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.2:36557)
0619 00:53:12.078208 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:12.078212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:12.078231 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:53:12.078348 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:12.078352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:12.078424 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:12.078506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:12.080126 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:12.081177 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:12.082989 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:12.083426 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:12.083523 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:12.088425 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:12.088444 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:12.088465 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:12.088516 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:12.092179 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:12.092991 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:12.093002 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:12.093006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:12.093174 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:12.093502 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:12.093507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:12.093511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:12.093966 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:12.094285 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:12.094461 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 00:53:12.094681 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 00:53:12.095860 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:12.207410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4712
I20260619 00:53:12.226733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4844
Jun 19 00:53:12 dist-test-slave-r7bh krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:53:12.243327 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:12.227500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.97.3:45741)
0619 00:53:12.227686 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:12.227690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:12.227761 (+ 71us) server_negotiation.cc:407] Connection header received
0619 00:53:12.227903 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:12.227908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:12.227969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:12.228055 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:12.229320 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:12.230022 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:12.231189 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:12.231622 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:12.231724 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:12.237228 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:12.237252 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:12.237276 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:12.237343 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:12.240288 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:12.241148 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:12.241156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:12.241159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:12.241303 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:12.241795 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:12.241800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:12.241802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:12.242223 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:12.242468 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:12.242874 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 00:53:12.243041 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:53:12.244002 4966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:12.333400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4844
I20260619 00:53:12.349491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 4487
I20260619 00:53:12.350697 4548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:12.458946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 4487
2026-06-19T00:53:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4413 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T00:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:12Z Disabled control of system clock
I20260619 00:53:12.527139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38093
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 00:53:12.864569 5023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:12.865000 5023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:12.865110 5023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:12.872538 5023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:12.872656 5023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:12.872694 5023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:12.872741 5023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:12.872787 5023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:12.882000 5023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:12.883222 5023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:12.884986 5023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:12.896445 5028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:12.896365 5029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:12.896402 5031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:12.897353 5023 server_base.cc:1034] running on GCE node
I20260619 00:53:12.898221 5023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:12.899616 5023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:12.900915 5023 hybrid_clock.cc:648] HybridClock initialized: now 1781830392900832 us; error 54 us; skew 500 ppm
I20260619 00:53:12.901297 5023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:12.907560 5023 webserver.cc:460] Webserver started at http://127.0.97.62:36973/ using document root <none> and password file <none>
I20260619 00:53:12.908305 5023 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:12.908406 5023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:12.908746 5023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:12.912016 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "e98dbeb0de844b05bd1a2fe7d713f933"
format_stamp: "Formatted at 2026-06-19 00:53:12 on dist-test-slave-r7bh"
server_key: "3df26fb2a98124917e5f6538fac7f8a7"
server_key_iv: "26a0cefc9914aec258514a1e84f02e8f"
server_key_version: "encryptionkey@0"
I20260619 00:53:12.912955 5023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "e98dbeb0de844b05bd1a2fe7d713f933"
format_stamp: "Formatted at 2026-06-19 00:53:12 on dist-test-slave-r7bh"
server_key: "3df26fb2a98124917e5f6538fac7f8a7"
server_key_iv: "26a0cefc9914aec258514a1e84f02e8f"
server_key_version: "encryptionkey@0"
I20260619 00:53:12.918717 5023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:12.922430 5037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:12.923905 5023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:12.924103 5023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "e98dbeb0de844b05bd1a2fe7d713f933"
format_stamp: "Formatted at 2026-06-19 00:53:12 on dist-test-slave-r7bh"
server_key: "3df26fb2a98124917e5f6538fac7f8a7"
server_key_iv: "26a0cefc9914aec258514a1e84f02e8f"
server_key_version: "encryptionkey@0"
I20260619 00:53:12.924353 5023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:12.930400 5023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:12.931192 5023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:12.931490 5023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:12.949808 5023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38093
I20260619 00:53:12.949849 5088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38093 every 8 connection(s)
I20260619 00:53:12.951256 5023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:12.954999 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:12.957345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5023
I20260619 00:53:12.957536 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:12.958329 388 external_mini_cluster.cc:1428] Setting key 17d8459883ab0ebb54754f12d0edd28d
I20260619 00:53:12.967366 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:12.960590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.0.97.62:38093)
0619 00:53:12.961664 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:12.961677 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:12.961927 (+ 250us) server_negotiation.cc:407] Connection header received
0619 00:53:12.963617 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:12.963642 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:12.964079 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:12.964521 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:12.965701 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:12.965707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:12.965724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:12.965738 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:12.965793 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:12.965807 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:12.966062 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:12.966200 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 00:53:12.966446 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":386,"threads_started":1}
I20260619 00:53:12.967717 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: Bootstrap starting.
I20260619 00:53:12.970985 5089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:12.972007 5089 log.cc:824] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:12.974985 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: No bootstrap required, opened a new log
I20260619 00:53:12.982623 5089 raft_consensus.cc:348] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } }
I20260619 00:53:12.983078 5089 raft_consensus.cc:374] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:12.983188 5089 raft_consensus.cc:729] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e98dbeb0de844b05bd1a2fe7d713f933, State: Initialized, Role: FOLLOWER
I20260619 00:53:12.983783 5089 consensus_queue.cc:260] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } }
I20260619 00:53:12.984037 5089 raft_consensus.cc:388] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:12.984191 5089 raft_consensus.cc:482] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:12.984362 5089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:12.986600 5089 raft_consensus.cc:504] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } }
I20260619 00:53:12.987434 5089 leader_election.cc:302] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e98dbeb0de844b05bd1a2fe7d713f933; no voters:
I20260619 00:53:12.988106 5089 leader_election.cc:288] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:12.988219 5094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:12.988960 5094 raft_consensus.cc:686] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [term 1 LEADER]: Becoming Leader. State: Replica: e98dbeb0de844b05bd1a2fe7d713f933, State: Running, Role: LEADER
I20260619 00:53:12.989569 5094 consensus_queue.cc:237] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } }
I20260619 00:53:12.990237 5089 sys_catalog.cc:564] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:12.992340 5096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e98dbeb0de844b05bd1a2fe7d713f933. Latest consensus state: current_term: 1 leader_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } } }
I20260619 00:53:12.992444 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98dbeb0de844b05bd1a2fe7d713f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38093 } } }
I20260619 00:53:12.992763 5096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:12.992761 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:12.994836 5103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:13.000161 5103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:13.009969 5103 catalog_manager.cc:1347] Generated new cluster ID: 2eebf04d37a04880b62bc2df6e729ed1
I20260619 00:53:13.010224 5103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:13.025413 5103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:13.025978 5103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:13.031529 5103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: Generated new TSK 0
I20260619 00:53:13.032616 5103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:53:13.040620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--builtin_ntp_servers=127.0.97.20:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 00:53:13.365218 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:13.365548 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:13.365645 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:13.372977 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:13.373090 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:13.373199 5113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:13.382550 5113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:13.383802 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:13.385591 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:13.400168 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:13.400236 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:13.400168 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:13.401289 5113 server_base.cc:1034] running on GCE node
I20260619 00:53:13.402014 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:13.403102 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:13.404420 5113 hybrid_clock.cc:648] HybridClock initialized: now 1781830393404353 us; error 58 us; skew 500 ppm
I20260619 00:53:13.404910 5113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:13.407541 5113 webserver.cc:460] Webserver started at http://127.0.97.1:45117/ using document root <none> and password file <none>
I20260619 00:53:13.408315 5113 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:13.408460 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:13.408928 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:13.412004 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "ac1d242483e34076b7e667182bd6f232"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "3e0bb8401a205aa83f637ec1cb97c1ed"
server_key_iv: "cc206eec81c327851731dc77f721eeaa"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.412957 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ac1d242483e34076b7e667182bd6f232"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "3e0bb8401a205aa83f637ec1cb97c1ed"
server_key_iv: "cc206eec81c327851731dc77f721eeaa"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.418473 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 00:53:13.422550 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.424111 5113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:53:13.424321 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "ac1d242483e34076b7e667182bd6f232"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "3e0bb8401a205aa83f637ec1cb97c1ed"
server_key_iv: "cc206eec81c327851731dc77f721eeaa"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.424594 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:13.431028 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:13.431922 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:13.432214 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:13.433352 5113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:13.434904 5113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:13.435032 5113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.435163 5113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:13.435248 5113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.469058 5113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44269
I20260619 00:53:13.469107 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44269 every 8 connection(s)
I20260619 00:53:13.470424 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:13.478790 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:13.473090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60563 (local address 127.0.97.62:38093)
0619 00:53:13.473643 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:13.473647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:13.474884 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 00:53:13.476548 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:13.476553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:13.476620 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:13.476756 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:13.478033 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:13.478039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:13.478043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:13.478068 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:13.478113 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:13.478125 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:13.478253 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:13.478438 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 00:53:13.478504 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":280,"threads_started":1}
I20260619 00:53:13.479777 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:13.473807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38093 (local address 127.0.97.1:60563)
0619 00:53:13.474714 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:13.474760 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:13.476289 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:13.477023 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:13.477032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:13.477128 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:13.477741 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:13.477743 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:13.477780 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:13.477861 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:13.478221 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:13.478230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:13.478238 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 00:53:13.478521 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 00:53:13.478791 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260619 00:53:13.480558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5113
I20260619 00:53:13.480773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:13.481632 5240 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38093
I20260619 00:53:13.481772 388 external_mini_cluster.cc:1428] Setting key 1421926a300a7082154954ebe1bdebc7
I20260619 00:53:13.481948 5240 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:13.482604 5240 heartbeater.cc:511] Master 127.0.97.62:38093 requested a full tablet report, sending...
I20260619 00:53:13.484741 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--builtin_ntp_servers=127.0.97.20:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 00:53:13.484735 5054 ts_manager.cc:194] Registered new tserver with Master: ac1d242483e34076b7e667182bd6f232 (127.0.97.1:44269)
W20260619 00:53:13.811594 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:13.811919 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:13.812088 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:13.819296 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:13.819445 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:13.819620 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:13.829020 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:13.830415 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:13.832266 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:13.845278 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:13.845324 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:13.845950 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:13.846504 5244 server_base.cc:1034] running on GCE node
I20260619 00:53:13.847091 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:13.848008 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:13.849198 5244 hybrid_clock.cc:648] HybridClock initialized: now 1781830393849139 us; error 63 us; skew 500 ppm
I20260619 00:53:13.849522 5244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:13.851706 5244 webserver.cc:460] Webserver started at http://127.0.97.2:33347/ using document root <none> and password file <none>
I20260619 00:53:13.852381 5244 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:13.852500 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:13.852859 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:13.855728 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "f824aab5eb3743f4ac796c7100393d30"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "30f4cc17484152439dc98f68c53c60f4"
server_key_iv: "adc08cebde5a6bd39e7b7b4e4749da13"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.856487 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f824aab5eb3743f4ac796c7100393d30"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "30f4cc17484152439dc98f68c53c60f4"
server_key_iv: "adc08cebde5a6bd39e7b7b4e4749da13"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.861477 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 00:53:13.864886 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.866207 5244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:53:13.866382 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "f824aab5eb3743f4ac796c7100393d30"
format_stamp: "Formatted at 2026-06-19 00:53:13 on dist-test-slave-r7bh"
server_key: "30f4cc17484152439dc98f68c53c60f4"
server_key_iv: "adc08cebde5a6bd39e7b7b4e4749da13"
server_key_version: "encryptionkey@0"
I20260619 00:53:13.866604 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:13.872088 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:13.872946 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:13.873207 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:13.874065 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:13.875416 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:13.875537 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.875662 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:13.875748 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:13.910159 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45709
I20260619 00:53:13.910214 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45709 every 8 connection(s)
I20260619 00:53:13.911551 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:13.914743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5244
I20260619 00:53:13.914916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:13.915544 388 external_mini_cluster.cc:1428] Setting key 1adee63d626b7869b7e3a542ef164ade
I20260619 00:53:13.918537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--builtin_ntp_servers=127.0.97.20:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 00:53:13.920885 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:13.914228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50581 (local address 127.0.97.62:38093)
0619 00:53:13.914398 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:13.914402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:13.916159 (+ 1757us) server_negotiation.cc:407] Connection header received
0619 00:53:13.918121 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:13.918129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:13.918203 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:13.918343 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:13.919840 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:13.919849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:13.919857 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:13.919883 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:13.919928 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:13.919959 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:13.920123 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:13.920384 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 00:53:13.920462 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:53:13.921813 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:13.914902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38093 (local address 127.0.97.2:50581)
0619 00:53:13.915961 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:13.916007 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:13.917811 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:13.918574 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:13.918583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:13.918752 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:13.919481 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:13.919483 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:13.919522 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:13.919609 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:13.920140 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:13.920151 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:13.920160 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 00:53:13.920422 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 00:53:13.920640 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":305,"threads_started":1}
I20260619 00:53:13.923834 5371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38093
I20260619 00:53:13.924189 5371 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:13.924789 5371 heartbeater.cc:511] Master 127.0.97.62:38093 requested a full tablet report, sending...
I20260619 00:53:13.926465 5054 ts_manager.cc:194] Registered new tserver with Master: f824aab5eb3743f4ac796c7100393d30 (127.0.97.2:45709)
W20260619 00:53:14.260174 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:14.260530 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:14.260694 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:14.268139 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:14.268301 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:14.268499 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:14.277868 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:14.279268 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:14.281147 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:14.294932 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:14.295161 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:14.295462 5374 server_base.cc:1034] running on GCE node
W20260619 00:53:14.295605 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:14.296479 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:14.297631 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:14.298854 5374 hybrid_clock.cc:648] HybridClock initialized: now 1781830394298806 us; error 53 us; skew 500 ppm
I20260619 00:53:14.299188 5374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:14.301712 5374 webserver.cc:460] Webserver started at http://127.0.97.3:42327/ using document root <none> and password file <none>
I20260619 00:53:14.302439 5374 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:14.302574 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:14.302922 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:14.306054 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "6a5b9335f805489d9b9ec23888b9537d"
format_stamp: "Formatted at 2026-06-19 00:53:14 on dist-test-slave-r7bh"
server_key: "f09eaee131f0b0d5d39c012905a53d49"
server_key_iv: "cdc695ea8ec21308bd44b78836dd785f"
server_key_version: "encryptionkey@0"
I20260619 00:53:14.306897 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6a5b9335f805489d9b9ec23888b9537d"
format_stamp: "Formatted at 2026-06-19 00:53:14 on dist-test-slave-r7bh"
server_key: "f09eaee131f0b0d5d39c012905a53d49"
server_key_iv: "cdc695ea8ec21308bd44b78836dd785f"
server_key_version: "encryptionkey@0"
I20260619 00:53:14.312631 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 00:53:14.317004 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:14.318670 5374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 00:53:14.318907 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "6a5b9335f805489d9b9ec23888b9537d"
format_stamp: "Formatted at 2026-06-19 00:53:14 on dist-test-slave-r7bh"
server_key: "f09eaee131f0b0d5d39c012905a53d49"
server_key_iv: "cdc695ea8ec21308bd44b78836dd785f"
server_key_version: "encryptionkey@0"
I20260619 00:53:14.319247 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:14.326416 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:14.327415 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:14.327872 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:14.329090 5374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:14.330762 5374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:14.330899 5374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:14.331073 5374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:14.331172 5374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:14.367388 5374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41589
I20260619 00:53:14.367434 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41589 every 8 connection(s)
I20260619 00:53:14.369009 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:14.370776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5374
I20260619 00:53:14.370945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:14.371449 388 external_mini_cluster.cc:1428] Setting key dab484cb1bda9afff9b62b032f8f1763
I20260619 00:53:14.379230 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.373424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33963 (local address 127.0.97.62:38093)
0619 00:53:14.373621 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.373625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.375301 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 00:53:14.377025 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.377030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.377100 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.377203 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.378516 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.378524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.378528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.378552 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.378581 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.378593 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.378722 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.378899 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.378966 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:53:14.380170 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.374077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38093 (local address 127.0.97.3:33963)
0619 00:53:14.375151 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.375191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.376708 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.377440 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.377448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.377549 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.378226 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.378228 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.378267 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.378353 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.378683 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.378693 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.378702 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.378983 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.379247 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":387,"threads_started":1}
I20260619 00:53:14.381640 5501 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38093
I20260619 00:53:14.381948 5501 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:14.382551 5501 heartbeater.cc:511] Master 127.0.97.62:38093 requested a full tablet report, sending...
I20260619 00:53:14.383948 5054 ts_manager.cc:194] Registered new tserver with Master: 6a5b9335f805489d9b9ec23888b9537d (127.0.97.3:41589)
I20260619 00:53:14.387084 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:53:14.394330 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.391525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.97.62:38093)
0619 00:53:14.391682 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.391705 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.392160 (+ 455us) server_negotiation.cc:407] Connection header received
0619 00:53:14.392363 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.392378 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.392430 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.392537 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.393647 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.393659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.393662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.393679 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.393740 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.393759 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.393900 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.393982 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.394045 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 00:53:14.398514 5054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:14.405257 5054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:14.435997 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.416947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43106)
0619 00:53:14.430397 (+ 13450us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.430464 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.430640 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.431837 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.431847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.431955 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.433528 (+ 1573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.433531 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.433565 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.433660 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.435123 (+ 1463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.435133 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.435141 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.435341 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.435444 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13098,"thread_start_us":11100,"threads_started":1}
I20260619 00:53:14.436573 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.417580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.97.3:41589)
0619 00:53:14.431576 (+ 13996us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.431584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.432662 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 00:53:14.433108 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.433116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.433343 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.433481 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.435055 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.435062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.435099 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.435118 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.435177 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.435197 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.435351 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.436085 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.436192 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13849,"thread_start_us":13709,"threads_started":1}
I20260619 00:53:14.436663 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.417030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41589 (local address 127.0.0.1:49222)
0619 00:53:14.432549 (+ 15519us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.432568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.432742 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.433855 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.433861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.433886 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.434859 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.434862 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.434874 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.434904 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.435404 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.435408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.435412 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.435554 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.435613 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15393,"thread_start_us":17276,"threads_started":1}
I20260619 00:53:14.437242 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.417094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.97.1:44269)
0619 00:53:14.431186 (+ 14092us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.431198 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.431234 (+ 36us) server_negotiation.cc:407] Connection header received
0619 00:53:14.431313 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.431321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.431597 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.432508 (+ 911us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.434843 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.434851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.434866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.434889 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.434942 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.434961 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.435148 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.436452 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.436545 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13929,"thread_start_us":13809,"threads_started":1}
I20260619 00:53:14.438040 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.417577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.0.1:43982)
0619 00:53:14.434551 (+ 16974us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.434570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.434710 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.435239 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.435244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.435266 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.436927 (+ 1661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.436930 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.436944 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.436986 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.437464 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.437468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.437473 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.437593 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.437640 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16856,"thread_start_us":14671,"threads_started":1}
I20260619 00:53:14.438619 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.418172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.0.97.2:45709)
0619 00:53:14.431993 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.432003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.434699 (+ 2696us) server_negotiation.cc:407] Connection header received
0619 00:53:14.434860 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.434868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.435079 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.435225 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.437150 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.437158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.437173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.437193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.437254 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.437275 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.437413 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.438338 (+ 925us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.438420 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13679,"thread_start_us":13569,"threads_started":1}
I20260619 00:53:14.441085 5436 tablet_service.cc:1467] Processing CreateTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 (DEFAULT_TABLE table=test-table [id=ba2cba46d8ec4e818583606ac9851ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:14.441083 5175 tablet_service.cc:1467] Processing CreateTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 (DEFAULT_TABLE table=test-table [id=ba2cba46d8ec4e818583606ac9851ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:14.441843 5306 tablet_service.cc:1467] Processing CreateTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 (DEFAULT_TABLE table=test-table [id=ba2cba46d8ec4e818583606ac9851ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:14.442580 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c6e39d9f864f80982e795bbe8b7289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:14.442582 5175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c6e39d9f864f80982e795bbe8b7289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:14.442988 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c6e39d9f864f80982e795bbe8b7289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:14.454190 5519 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Bootstrap starting.
I20260619 00:53:14.457520 5521 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Bootstrap starting.
I20260619 00:53:14.458056 5519 tablet_bootstrap.cc:654] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:14.459236 5520 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Bootstrap starting.
I20260619 00:53:14.459517 5519 log.cc:824] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:14.461036 5521 tablet_bootstrap.cc:654] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:14.462232 5521 log.cc:824] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:14.462373 5520 tablet_bootstrap.cc:654] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:14.462821 5519 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: No bootstrap required, opened a new log
I20260619 00:53:14.463156 5519 ts_tablet_manager.cc:1397] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 00:53:14.463490 5520 log.cc:824] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:14.467314 5521 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: No bootstrap required, opened a new log
I20260619 00:53:14.467496 5520 tablet_bootstrap.cc:492] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: No bootstrap required, opened a new log
I20260619 00:53:14.467640 5521 ts_tablet_manager.cc:1397] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260619 00:53:14.467763 5520 ts_tablet_manager.cc:1397] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260619 00:53:14.474072 5519 raft_consensus.cc:348] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.474514 5519 raft_consensus.cc:374] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:14.474596 5519 raft_consensus.cc:729] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f824aab5eb3743f4ac796c7100393d30, State: Initialized, Role: FOLLOWER
I20260619 00:53:14.475333 5519 consensus_queue.cc:260] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.476401 5521 raft_consensus.cc:348] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.476845 5521 raft_consensus.cc:374] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:14.476953 5521 raft_consensus.cc:729] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5b9335f805489d9b9ec23888b9537d, State: Initialized, Role: FOLLOWER
I20260619 00:53:14.477608 5521 consensus_queue.cc:260] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.482270 5520 raft_consensus.cc:348] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.484557 5520 raft_consensus.cc:374] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:14.484722 5520 raft_consensus.cc:729] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1d242483e34076b7e667182bd6f232, State: Initialized, Role: FOLLOWER
I20260619 00:53:14.485231 5501 heartbeater.cc:503] Master 127.0.97.62:38093 was elected leader, sending a full tablet report...
I20260619 00:53:14.485483 5520 consensus_queue.cc:260] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.485699 5521 ts_tablet_manager.cc:1428] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20260619 00:53:14.486755 5240 heartbeater.cc:503] Master 127.0.97.62:38093 was elected leader, sending a full tablet report...
I20260619 00:53:14.487275 5520 ts_tablet_manager.cc:1428] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Time spent starting tablet: real 0.019s user 0.016s sys 0.001s
I20260619 00:53:14.491454 5371 heartbeater.cc:503] Master 127.0.97.62:38093 was elected leader, sending a full tablet report...
I20260619 00:53:14.492215 5519 ts_tablet_manager.cc:1428] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Time spent starting tablet: real 0.029s user 0.009s sys 0.018s
W20260619 00:53:14.621124 5502 tablet.cc:2369] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:14.663699 5372 tablet.cc:2369] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:14.722846 5241 tablet.cc:2369] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:14.781586 5526 raft_consensus.cc:482] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:14.781924 5526 raft_consensus.cc:504] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.783689 5526 leader_election.cc:288] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac1d242483e34076b7e667182bd6f232 (127.0.97.1:44269), f824aab5eb3743f4ac796c7100393d30 (127.0.97.2:45709)
I20260619 00:53:14.786284 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.783978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45709 (local address 127.0.97.3:40093)
0619 00:53:14.784210 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.784233 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.784347 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.784711 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.784715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.784741 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.785442 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.785444 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.785461 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.785503 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.785861 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.785867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.785873 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.785984 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.786042 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 00:53:14.786326 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.784175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40093 (local address 127.0.97.2:45709)
0619 00:53:14.784350 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.784354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.784371 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:14.784468 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.784475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.784543 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.784664 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.785656 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.785663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.785669 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.785692 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.785730 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.785759 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.785874 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.785984 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.786052 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:53:14.787017 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.783978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34915 (local address 127.0.97.1:44269)
0619 00:53:14.784210 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.784217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.785046 (+ 829us) server_negotiation.cc:407] Connection header received
0619 00:53:14.785189 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.785194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.785250 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.785348 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.786334 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.786340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.786344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.786363 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.786398 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.786415 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.786530 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.786693 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.786749 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 00:53:14.787199 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.784036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.97.3:34915)
0619 00:53:14.784946 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:14.784966 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:14.785083 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:14.785368 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:14.785373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:14.785399 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:14.786121 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:53:14.786125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:14.786142 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:53:14.786185 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:14.786571 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:14.786581 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:14.786586 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 00:53:14.786704 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 00:53:14.786748 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":91,"thread_start_us":404,"threads_started":1}
I20260619 00:53:14.787225 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c6e39d9f864f80982e795bbe8b7289" candidate_uuid: "6a5b9335f805489d9b9ec23888b9537d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f824aab5eb3743f4ac796c7100393d30" is_pre_election: true
I20260619 00:53:14.787771 5326 raft_consensus.cc:2454] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a5b9335f805489d9b9ec23888b9537d in term 0.
I20260619 00:53:14.788049 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c6e39d9f864f80982e795bbe8b7289" candidate_uuid: "6a5b9335f805489d9b9ec23888b9537d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1d242483e34076b7e667182bd6f232" is_pre_election: true
I20260619 00:53:14.788479 5392 leader_election.cc:302] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5b9335f805489d9b9ec23888b9537d, f824aab5eb3743f4ac796c7100393d30; no voters:
I20260619 00:53:14.788594 5195 raft_consensus.cc:2454] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a5b9335f805489d9b9ec23888b9537d in term 0.
I20260619 00:53:14.788869 5526 raft_consensus.cc:2781] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:14.789031 5526 raft_consensus.cc:482] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:14.789139 5526 raft_consensus.cc:3037] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:14.791479 5526 raft_consensus.cc:504] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.792080 5526 leader_election.cc:288] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [CANDIDATE]: Term 1 election: Requested vote from peers ac1d242483e34076b7e667182bd6f232 (127.0.97.1:44269), f824aab5eb3743f4ac796c7100393d30 (127.0.97.2:45709)
I20260619 00:53:14.792490 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c6e39d9f864f80982e795bbe8b7289" candidate_uuid: "6a5b9335f805489d9b9ec23888b9537d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f824aab5eb3743f4ac796c7100393d30"
I20260619 00:53:14.792471 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c6e39d9f864f80982e795bbe8b7289" candidate_uuid: "6a5b9335f805489d9b9ec23888b9537d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1d242483e34076b7e667182bd6f232"
I20260619 00:53:14.792681 5195 raft_consensus.cc:3037] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:14.792681 5326 raft_consensus.cc:3037] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:14.795902 5326 raft_consensus.cc:2454] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a5b9335f805489d9b9ec23888b9537d in term 1.
I20260619 00:53:14.795904 5195 raft_consensus.cc:2454] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a5b9335f805489d9b9ec23888b9537d in term 1.
I20260619 00:53:14.796399 5389 leader_election.cc:302] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5b9335f805489d9b9ec23888b9537d, ac1d242483e34076b7e667182bd6f232; no voters:
I20260619 00:53:14.796722 5526 raft_consensus.cc:2781] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:14.797186 5526 raft_consensus.cc:686] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 1 LEADER]: Becoming Leader. State: Replica: 6a5b9335f805489d9b9ec23888b9537d, State: Running, Role: LEADER
I20260619 00:53:14.797745 5526 consensus_queue.cc:237] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } }
I20260619 00:53:14.802664 5053 catalog_manager.cc:5495] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a5b9335f805489d9b9ec23888b9537d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6a5b9335f805489d9b9ec23888b9537d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a5b9335f805489d9b9ec23888b9537d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 } health_report { overall_health: UNKNOWN } } }
I20260619 00:53:14.896078 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.893799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.97.3:41589)
0619 00:53:14.894006 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.894013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.894118 (+ 105us) server_negotiation.cc:407] Connection header received
0619 00:53:14.894378 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.894382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.894462 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.894593 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.895478 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.895484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.895489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.895508 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.895544 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.895555 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.895685 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.895778 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.895834 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 00:53:14.903982 5326 raft_consensus.cc:1261] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 1 FOLLOWER]: Refusing update from remote peer 6a5b9335f805489d9b9ec23888b9537d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:14.903996 5195 raft_consensus.cc:1261] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 1 FOLLOWER]: Refusing update from remote peer 6a5b9335f805489d9b9ec23888b9537d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:14.904933 5529 consensus_queue.cc:1035] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1d242483e34076b7e667182bd6f232" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:14.905282 5526 consensus_queue.cc:1035] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f824aab5eb3743f4ac796c7100393d30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:14.917834 5532 mvcc.cc:205] Tried to move back new op lower bound from 7298377297517662208 to 7298377297096208384. Current Snapshot: MvccSnapshot[applied={T|T < 7298377297517662208}]
I20260619 00:53:14.918689 5533 mvcc.cc:205] Tried to move back new op lower bound from 7298377297517662208 to 7298377297096208384. Current Snapshot: MvccSnapshot[applied={T|T < 7298377297517662208}]
I20260619 00:53:14.920662 5534 mvcc.cc:205] Tried to move back new op lower bound from 7298377297517662208 to 7298377297096208384. Current Snapshot: MvccSnapshot[applied={T|T < 7298377297517662208}]
I20260619 00:53:14.939373 5053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:14.939653 5053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:14.943272 5053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e98dbeb0de844b05bd1a2fe7d713f933: Sending DeleteTablet for 3 replicas of tablet a5c6e39d9f864f80982e795bbe8b7289
I20260619 00:53:14.944448 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:14 UTC) from {username='slave'} at 127.0.0.1:49222
I20260619 00:53:14.944442 5175 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:14 UTC) from {username='slave'} at 127.0.0.1:43106
I20260619 00:53:14.944504 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet a5c6e39d9f864f80982e795bbe8b7289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:14 UTC) from {username='slave'} at 127.0.0.1:43982
I20260619 00:53:14.947669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5113
I20260619 00:53:14.954911 5544 tablet_replica.cc:330] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: stopping tablet replica
I20260619 00:53:14.955435 5544 raft_consensus.cc:2229] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:14.955950 5544 raft_consensus.cc:2258] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:14.956796 5543 tablet_replica.cc:330] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: stopping tablet replica
I20260619 00:53:14.957233 5543 raft_consensus.cc:2229] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:14.957705 5543 raft_consensus.cc:2258] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:14.959985 5543 ts_tablet_manager.cc:1905] T a5c6e39d9f864f80982e795bbe8b7289 P ac1d242483e34076b7e667182bd6f232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:14.963960 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:14.949089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.0.97.1:44269)
0619 00:53:14.949256 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:14.949262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:14.961348 (+ 12086us) server_negotiation.cc:407] Connection header received
0619 00:53:14.961643 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:14.961649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:14.961741 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:14.961818 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:14.963017 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:14.963026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:14.963034 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:14.963058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:14.963095 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:14.963111 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:14.963228 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:14.963645 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 00:53:14.963703 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:53:14.964772 5235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:14.965601 5544 ts_tablet_manager.cc:1905] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:14.970299 5542 tablet_replica.cc:330] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: stopping tablet replica
I20260619 00:53:14.970875 5542 raft_consensus.cc:2229] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:14.971597 5542 raft_consensus.cc:2258] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:14.973896 5542 ts_tablet_manager.cc:1905] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:14.979941 5544 ts_tablet_manager.cc:1918] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:14.980144 5544 log.cc:1196] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/a5c6e39d9f864f80982e795bbe8b7289
I20260619 00:53:14.980661 5544 ts_tablet_manager.cc:1939] T a5c6e39d9f864f80982e795bbe8b7289 P f824aab5eb3743f4ac796c7100393d30: Deleting consensus metadata
I20260619 00:53:14.984282 5041 catalog_manager.cc:4841] TS f824aab5eb3743f4ac796c7100393d30 (127.0.97.2:45709): tablet a5c6e39d9f864f80982e795bbe8b7289 (table test-table [id=ba2cba46d8ec4e818583606ac9851ec3]) successfully deleted
I20260619 00:53:14.987154 5542 ts_tablet_manager.cc:1918] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:14.987371 5542 log.cc:1196] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/a5c6e39d9f864f80982e795bbe8b7289
I20260619 00:53:14.987861 5542 ts_tablet_manager.cc:1939] T a5c6e39d9f864f80982e795bbe8b7289 P 6a5b9335f805489d9b9ec23888b9537d: Deleting consensus metadata
I20260619 00:53:14.989857 5040 catalog_manager.cc:4841] TS 6a5b9335f805489d9b9ec23888b9537d (127.0.97.3:41589): tablet a5c6e39d9f864f80982e795bbe8b7289 (table test-table [id=ba2cba46d8ec4e818583606ac9851ec3]) successfully deleted
I20260619 00:53:15.094765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5113
W20260619 00:53:15.111645 5038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44269 (error 108)
I20260619 00:53:15.112987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5244
I20260619 00:53:15.113867 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.112942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43122)
0619 00:53:15.113504 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.113651 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":310,"threads_started":1}
W20260619 00:53:15.114239 5038 catalog_manager.cc:4584] TS ac1d242483e34076b7e667182bd6f232 (127.0.97.1:44269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5c6e39d9f864f80982e795bbe8b7289: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
I20260619 00:53:15.116091 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.113778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.97.2:45709)
0619 00:53:15.114006 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:15.114013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:15.114072 (+ 59us) server_negotiation.cc:407] Connection header received
0619 00:53:15.114296 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:15.114301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:15.114413 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:15.114522 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:15.115462 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:15.115467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:15.115470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:15.115484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:15.115513 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:15.115525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:15.115640 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:15.115734 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 00:53:15.115779 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 00:53:15.116722 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:15.168198 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.167606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43130)
0619 00:53:15.167868 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.168021 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 00:53:15.169417 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.168995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43132)
0619 00:53:15.169174 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.169259 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 00:53:15.205526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5244
I20260619 00:53:15.222380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5374
I20260619 00:53:15.225768 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.223256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.0.97.3:41589)
0619 00:53:15.223462 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:15.223470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:15.223568 (+ 98us) server_negotiation.cc:407] Connection header received
0619 00:53:15.223831 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:15.223835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:15.223884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:15.224035 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:15.225062 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:15.225071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:15.225076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:15.225093 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:15.225159 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:15.225171 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:15.225277 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:15.225401 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 00:53:15.225444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:53:15.226557 5496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:15.230186 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.229805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43144)
0619 00:53:15.229988 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.230055 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 00:53:15.231334 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.231003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43146)
0619 00:53:15.231145 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.231215 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 00:53:15.312000 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.311550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43148)
0619 00:53:15.311766 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.311858 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 00:53:15.313202 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.312733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44269 (local address 127.0.0.1:43162)
0619 00:53:15.312954 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:15.313060 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 00:53:15.346769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5374
I20260619 00:53:15.365073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5023
I20260619 00:53:15.366245 5084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:15.470718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5023
2026-06-19T00:53:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3009 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 00:53:15.511579 388 mini_oidc.cc:152] Starting JWKS server
I20260619 00:53:15.514155 388 webserver.cc:460] Webserver started at https://127.0.0.1:40405/ using document root <none> and password file <none>
I20260619 00:53:15.514427 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 00:53:15.515043 388 webserver.cc:460] Webserver started at http://127.0.0.1:35111/ using document root <none> and password file <none>
2026-06-19T00:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:15Z Disabled control of system clock
I20260619 00:53:15.542948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35787
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35787
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/testchainca.cert with env {}
W20260619 00:53:15.865763 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:15.866072 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:15.866160 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:15.873723 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:15.873848 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:15.873888 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:15.873937 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:15.873982 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 00:53:15.874049 5557 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 00:53:15.874089 5557 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40405/jwks.json
I20260619 00:53:15.884222 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:15.885495 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:15.886749 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:15.898824 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:15.898882 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:15.898904 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:15.899986 5557 server_base.cc:1034] running on GCE node
I20260619 00:53:15.900926 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:15.902400 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:15.903636 5557 hybrid_clock.cc:648] HybridClock initialized: now 1781830395903582 us; error 49 us; skew 500 ppm
I20260619 00:53:15.903992 5557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:15.906291 5557 webserver.cc:460] Webserver started at http://127.0.97.62:37019/ using document root <none> and password file <none>
I20260619 00:53:15.906967 5557 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:15.907109 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:15.907418 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:15.910346 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "79cb28f76b234363a02a54d12940c99a"
format_stamp: "Formatted at 2026-06-19 00:53:15 on dist-test-slave-r7bh"
server_key: "ac3af1cf20ad27d5f529ac8b0ee0d0b3"
server_key_iv: "9273f55c815d5a3a592aaf903f1190f3"
server_key_version: "encryptionkey@0"
I20260619 00:53:15.911194 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "79cb28f76b234363a02a54d12940c99a"
format_stamp: "Formatted at 2026-06-19 00:53:15 on dist-test-slave-r7bh"
server_key: "ac3af1cf20ad27d5f529ac8b0ee0d0b3"
server_key_iv: "9273f55c815d5a3a592aaf903f1190f3"
server_key_version: "encryptionkey@0"
I20260619 00:53:15.916461 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:53:15.919901 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:15.921540 5557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:53:15.921741 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "79cb28f76b234363a02a54d12940c99a"
format_stamp: "Formatted at 2026-06-19 00:53:15 on dist-test-slave-r7bh"
server_key: "ac3af1cf20ad27d5f529ac8b0ee0d0b3"
server_key_iv: "9273f55c815d5a3a592aaf903f1190f3"
server_key_version: "encryptionkey@0"
I20260619 00:53:15.921960 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:15.940550 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:15.941583 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:15.941876 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:15.960647 5557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35787
I20260619 00:53:15.960736 5622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35787 every 8 connection(s)
I20260619 00:53:15.962126 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:15.965940 5623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:15.972321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5557
I20260619 00:53:15.972550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:15.973296 388 external_mini_cluster.cc:1428] Setting key 8610dbe50a870dffdf0386a124cafa99
I20260619 00:53:15.976722 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a: Bootstrap starting.
I20260619 00:53:15.980018 5623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:15.981173 5623 log.cc:824] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:15.983927 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a: No bootstrap required, opened a new log
I20260619 00:53:15.987725 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:15.975352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.0.97.62:35787)
0619 00:53:15.976451 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:15.976464 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:15.976509 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:53:15.978435 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:15.978458 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:15.978949 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:15.979291 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:15.979969 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:15.981625 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:15.983104 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:15.983837 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:15.984060 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:15.985331 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:15.985338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:15.985353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:15.985937 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:15.986018 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:15.986040 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:15.986205 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:15.986458 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 00:53:15.986768 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":429,"threads_started":1}
I20260619 00:53:15.991919 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:53:15.992543 5623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } }
I20260619 00:53:15.992990 5623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:15.993098 5623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cb28f76b234363a02a54d12940c99a, State: Initialized, Role: FOLLOWER
I20260619 00:53:15.993690 5623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } }
I20260619 00:53:15.993921 5623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:15.994063 5623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:15.994222 5623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:15.996960 5623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } }
I20260619 00:53:15.997514 5623 leader_election.cc:302] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cb28f76b234363a02a54d12940c99a; no voters:
I20260619 00:53:15.998082 5623 leader_election.cc:288] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:15.998255 5628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:15.999053 5628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [term 1 LEADER]: Becoming Leader. State: Replica: 79cb28f76b234363a02a54d12940c99a, State: Running, Role: LEADER
I20260619 00:53:15.999603 5628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } }
I20260619 00:53:16.000216 5623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:16.002256 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cb28f76b234363a02a54d12940c99a. Latest consensus state: current_term: 1 leader_uuid: "79cb28f76b234363a02a54d12940c99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } } }
I20260619 00:53:16.002743 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:16.004640 5635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:16.004767 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79cb28f76b234363a02a54d12940c99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cb28f76b234363a02a54d12940c99a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35787 } } }
I20260619 00:53:16.005096 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:16.010469 5635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:16.020725 5635 catalog_manager.cc:1347] Generated new cluster ID: 720d2d1e9e584602b09377ecd52eca22
I20260619 00:53:16.020920 5635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:16.029395 5635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:16.030773 5635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:16.044277 5635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79cb28f76b234363a02a54d12940c99a: Generated new TSK 0
I20260619 00:53:16.045313 5635 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 00:53:16.093617 5653 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 00:53:16.093793 5653 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 00:53:16.094686 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:16.065987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.97.62:35787)
0619 00:53:16.066225 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:16.066234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:16.066684 (+ 450us) server_negotiation.cc:407] Connection header received
0619 00:53:16.067021 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:16.067025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:16.067096 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:16.067251 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0619 00:53:16.067797 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.068533 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.070032 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.070466 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.070608 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:16.071117 (+ 509us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 00:53:16.094021 (+ 22904us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 00:53:16.094400 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 00:53:16.094825 5626 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 00:53:16.096199 5652 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:16.065828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35787 (local address 127.0.0.1:38244)
0619 00:53:16.066554 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:16.066573 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:16.066777 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:16.067275 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:16.067280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:16.067292 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 00:53:16.067636 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:16.067644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.068681 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.068688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:16.069880 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:16.069891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.070595 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.070662 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:16.070802 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:16.070855 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 00:53:16.094321 (+ 23466us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 00:53:16.094684 (+ 363us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":425,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 00:53:16.097839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5557
I20260619 00:53:16.099009 5618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:16.191468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5557
2026-06-19T00:53:16Z chronyd exiting
I20260619 00:53:16.241801 388 test_util.cc:175] -----------------------------------------------
I20260619 00:53:16.241977 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830351410501-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (734 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 00:53:16.245749 388 mini_oidc.cc:152] Starting JWKS server
I20260619 00:53:16.247646 388 webserver.cc:460] Webserver started at https://127.0.0.1:34161/ using document root <none> and password file <none>
I20260619 00:53:16.247893 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 00:53:16.248631 388 webserver.cc:460] Webserver started at http://127.0.0.1:34779/ using document root <none> and password file <none>
2026-06-19T00:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:16Z Disabled control of system clock
I20260619 00:53:16.276423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39375
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34161/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/testchainca.cert with env {}
W20260619 00:53:16.600082 5660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:16.600380 5660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:16.600538 5660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:16.608078 5660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:16.608199 5660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:16.608275 5660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:16.608383 5660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:16.608444 5660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 00:53:16.608567 5660 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 00:53:16.608633 5660 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34161/jwks.json
I20260619 00:53:16.619030 5660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34161/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:16.620342 5660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:16.621692 5660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:16.633782 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:16.633819 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:16.633769 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:16.634842 5660 server_base.cc:1034] running on GCE node
I20260619 00:53:16.635663 5660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:16.637238 5660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:16.638486 5660 hybrid_clock.cc:648] HybridClock initialized: now 1781830396638448 us; error 33 us; skew 500 ppm
I20260619 00:53:16.638881 5660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:16.641396 5660 webserver.cc:460] Webserver started at http://127.0.97.62:42527/ using document root <none> and password file <none>
I20260619 00:53:16.642115 5660 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:16.642256 5660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:16.642593 5660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:16.645622 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "05b40d47818b47edb771bec40fab3bd6"
format_stamp: "Formatted at 2026-06-19 00:53:16 on dist-test-slave-r7bh"
server_key: "7409b2243c359a63415b93226b948915"
server_key_iv: "c23727beb022edd0f0c4645d069d20bc"
server_key_version: "encryptionkey@0"
I20260619 00:53:16.646486 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "05b40d47818b47edb771bec40fab3bd6"
format_stamp: "Formatted at 2026-06-19 00:53:16 on dist-test-slave-r7bh"
server_key: "7409b2243c359a63415b93226b948915"
server_key_iv: "c23727beb022edd0f0c4645d069d20bc"
server_key_version: "encryptionkey@0"
I20260619 00:53:16.652016 5660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:16.655658 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:16.657196 5660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:16.657392 5660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "05b40d47818b47edb771bec40fab3bd6"
format_stamp: "Formatted at 2026-06-19 00:53:16 on dist-test-slave-r7bh"
server_key: "7409b2243c359a63415b93226b948915"
server_key_iv: "c23727beb022edd0f0c4645d069d20bc"
server_key_version: "encryptionkey@0"
I20260619 00:53:16.657618 5660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:16.678045 5660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:16.678951 5660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:16.679257 5660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:16.698235 5660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39375
I20260619 00:53:16.698263 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39375 every 8 connection(s)
I20260619 00:53:16.699759 5660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:16.703910 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:16.705278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5660
I20260619 00:53:16.705482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:16.706287 388 external_mini_cluster.cc:1428] Setting key 5e23980e161fb0496b71b90841bea33f
I20260619 00:53:16.717324 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6: Bootstrap starting.
I20260619 00:53:16.719666 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:16.708311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.0.97.62:39375)
0619 00:53:16.709431 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:16.709443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:16.709487 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:53:16.711025 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:16.711048 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:16.711518 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:16.711932 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:16.712480 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.713961 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.715204 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.715937 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.716123 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:16.717369 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:16.717377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:16.717391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:16.717962 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:16.718039 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:16.718059 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:16.718185 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:16.718413 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 00:53:16.718745 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":408,"threads_started":1}
I20260619 00:53:16.720615 5726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:16.721961 5726 log.cc:824] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:16.723718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:53:16.724957 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6: No bootstrap required, opened a new log
I20260619 00:53:16.732544 5726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } }
I20260619 00:53:16.732937 5726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:16.733019 5726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b40d47818b47edb771bec40fab3bd6, State: Initialized, Role: FOLLOWER
I20260619 00:53:16.733515 5726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } }
I20260619 00:53:16.733711 5726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:16.733803 5726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:16.733919 5726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:16.736193 5726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } }
I20260619 00:53:16.736732 5726 leader_election.cc:302] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b40d47818b47edb771bec40fab3bd6; no voters:
I20260619 00:53:16.737345 5726 leader_election.cc:288] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:16.737501 5731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:16.738505 5731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 05b40d47818b47edb771bec40fab3bd6, State: Running, Role: LEADER
I20260619 00:53:16.739013 5731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } }
I20260619 00:53:16.739293 5726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:16.741704 5733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b40d47818b47edb771bec40fab3bd6. Latest consensus state: current_term: 1 leader_uuid: "05b40d47818b47edb771bec40fab3bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } } }
I20260619 00:53:16.741818 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b40d47818b47edb771bec40fab3bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b40d47818b47edb771bec40fab3bd6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39375 } } }
I20260619 00:53:16.742098 5733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:16.742183 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:16.745121 5739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:16.749250 5739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:16.757123 5739 catalog_manager.cc:1347] Generated new cluster ID: 2dc52460f4e741248db02e65ae548634
I20260619 00:53:16.757241 5739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:16.772217 5739 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:16.774168 5739 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:16.790504 5739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b40d47818b47edb771bec40fab3bd6: Generated new TSK 0
I20260619 00:53:16.791649 5739 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 00:53:16.825567 5756 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 00:53:16.825719 5756 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 00:53:16.826840 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:16.806626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.0.97.62:39375)
0619 00:53:16.806845 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:16.806853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:16.807311 (+ 458us) server_negotiation.cc:407] Connection header received
0619 00:53:16.807537 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:16.807541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:16.807616 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:16.807760 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0619 00:53:16.808329 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.809181 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.811183 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.811653 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.811751 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:16.812346 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 00:53:16.826227 (+ 13881us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 00:53:16.826552 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 00:53:16.826977 5729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 00:53:16.827554 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:16.806478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39375 (local address 127.0.0.1:59236)
0619 00:53:16.807179 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:16.807200 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:16.807366 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:16.807807 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:16.807813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:16.807826 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 00:53:16.808160 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:16.808171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.809326 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.809332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:16.811031 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:16.811046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:16.811774 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:16.811824 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:16.812030 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:16.812077 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 00:53:16.826498 (+ 14421us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 00:53:16.826795 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":362,"threads_started":1}
I20260619 00:53:16.829175 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5660
I20260619 00:53:16.830399 5721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:16.923125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5660
2026-06-19T00:53:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 00:53:16.979216 388 mini_oidc.cc:152] Starting JWKS server
I20260619 00:53:16.981089 388 webserver.cc:460] Webserver started at https://127.0.0.1:37977/ using document root <none> and password file <none>
I20260619 00:53:16.981297 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 00:53:16.981885 388 webserver.cc:460] Webserver started at http://127.0.0.1:45625/ using document root <none> and password file <none>
2026-06-19T00:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:16Z Disabled control of system clock
I20260619 00:53:17.009217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39305
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39305
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 00:53:17.331079 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:17.331391 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:17.331540 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:17.338713 5763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:17.338817 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:17.338889 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:17.339001 5763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:17.339062 5763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 00:53:17.339143 5763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 00:53:17.339218 5763 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37977/jwks.json
I20260619 00:53:17.348371 5763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:17.349731 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:17.351496 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:17.363376 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:17.363421 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:17.363389 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:17.364331 5763 server_base.cc:1034] running on GCE node
I20260619 00:53:17.365258 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:17.366735 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:17.367974 5763 hybrid_clock.cc:648] HybridClock initialized: now 1781830397367929 us; error 42 us; skew 500 ppm
I20260619 00:53:17.368328 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:17.370820 5763 webserver.cc:460] Webserver started at http://127.0.97.62:36573/ using document root <none> and password file <none>
I20260619 00:53:17.371577 5763 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:17.371718 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:17.372059 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:17.375066 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "8388e3acf7944de6b8d3d023a5e93945"
format_stamp: "Formatted at 2026-06-19 00:53:17 on dist-test-slave-r7bh"
server_key: "b2d9e0355d7ec36b72ca7b4b3cae4a0c"
server_key_iv: "18da0b817305fa6f7a8462c31f87eabd"
server_key_version: "encryptionkey@0"
I20260619 00:53:17.375874 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "8388e3acf7944de6b8d3d023a5e93945"
format_stamp: "Formatted at 2026-06-19 00:53:17 on dist-test-slave-r7bh"
server_key: "b2d9e0355d7ec36b72ca7b4b3cae4a0c"
server_key_iv: "18da0b817305fa6f7a8462c31f87eabd"
server_key_version: "encryptionkey@0"
I20260619 00:53:17.381177 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 00:53:17.384899 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:17.386312 5763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:17.386510 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "8388e3acf7944de6b8d3d023a5e93945"
format_stamp: "Formatted at 2026-06-19 00:53:17 on dist-test-slave-r7bh"
server_key: "b2d9e0355d7ec36b72ca7b4b3cae4a0c"
server_key_iv: "18da0b817305fa6f7a8462c31f87eabd"
server_key_version: "encryptionkey@0"
I20260619 00:53:17.386742 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:17.434286 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:17.435173 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:17.435478 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:17.454190 5763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39305
I20260619 00:53:17.454234 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39305 every 8 connection(s)
I20260619 00:53:17.455577 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:17.459753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5763
I20260619 00:53:17.459981 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:17.460158 5829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:17.460907 388 external_mini_cluster.cc:1428] Setting key 98f3ca1f7754e94158e0516116846026
I20260619 00:53:17.473634 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945: Bootstrap starting.
I20260619 00:53:17.474668 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:17.463025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.0.97.62:39305)
0619 00:53:17.464073 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:17.464087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:17.464128 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:53:17.465759 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:17.465781 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:17.466279 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:17.466712 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:17.467309 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.468576 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.469985 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.470717 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.470948 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:17.472157 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:17.472165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:17.472182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:17.472738 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:17.472895 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:17.472919 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:17.473042 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:17.473290 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 00:53:17.473619 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":351,"threads_started":1}
I20260619 00:53:17.477051 5829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:17.478209 5829 log.cc:824] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:17.478643 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:53:17.481556 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945: No bootstrap required, opened a new log
I20260619 00:53:17.488783 5829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } }
I20260619 00:53:17.489269 5829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:17.489389 5829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8388e3acf7944de6b8d3d023a5e93945, State: Initialized, Role: FOLLOWER
I20260619 00:53:17.490032 5829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } }
I20260619 00:53:17.490244 5829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:17.490409 5829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:17.490538 5829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:17.492734 5829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } }
I20260619 00:53:17.493265 5829 leader_election.cc:302] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8388e3acf7944de6b8d3d023a5e93945; no voters:
I20260619 00:53:17.493832 5829 leader_election.cc:288] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:17.493994 5834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:17.494725 5834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [term 1 LEADER]: Becoming Leader. State: Replica: 8388e3acf7944de6b8d3d023a5e93945, State: Running, Role: LEADER
I20260619 00:53:17.495317 5834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } }
I20260619 00:53:17.496227 5829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:17.498528 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8388e3acf7944de6b8d3d023a5e93945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } } }
I20260619 00:53:17.498507 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8388e3acf7944de6b8d3d023a5e93945. Latest consensus state: current_term: 1 leader_uuid: "8388e3acf7944de6b8d3d023a5e93945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8388e3acf7944de6b8d3d023a5e93945" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39305 } } }
I20260619 00:53:17.498950 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:17.499010 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:17.500479 5840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:17.505888 5840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:17.515422 5840 catalog_manager.cc:1347] Generated new cluster ID: dd543a19dec14eefbc64b87807238765
I20260619 00:53:17.515542 5840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:17.526742 5840 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:17.528092 5840 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:17.539215 5840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8388e3acf7944de6b8d3d023a5e93945: Generated new TSK 0
I20260619 00:53:17.540146 5840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:53:17.564414 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:17.554459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.0.97.62:39305)
0619 00:53:17.554716 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:17.554725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:17.555284 (+ 559us) server_negotiation.cc:407] Connection header received
0619 00:53:17.555585 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:17.555590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:17.555674 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:17.555832 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0619 00:53:17.556492 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.557812 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.559941 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.560425 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.560555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:17.561282 (+ 727us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 00:53:17.563749 (+ 2467us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 00:53:17.564097 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 00:53:17.564572 5832 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 00:53:17.566609 5858 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:17.554392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39305 (local address 127.0.0.1:32966)
0619 00:53:17.555131 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:17.555153 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:17.555335 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:17.555897 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:17.555904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:17.555919 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0619 00:53:17.556291 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:17.556304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.558015 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.558022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:17.559764 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:17.559781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:17.560567 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:17.560650 (+ 83us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:17.560887 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:17.560942 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 00:53:17.564001 (+ 3059us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 00:53:17.566127 (+ 2126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39305: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":1766,"thread_start_us":384,"threads_started":1}
I20260619 00:53:17.568902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5763
I20260619 00:53:17.570139 5824 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 00:53:17.576315 5859 webserver.cc:533] Webserver: sslize failed:
E20260619 00:53:17.576449 5859 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 00:53:17.660789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5763
2026-06-19T00:53:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T00:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:17Z Disabled control of system clock
I20260619 00:53:17.738722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34327
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34327
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/testchainca.cert with env {}
W20260619 00:53:18.063614 5864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:18.063917 5864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:18.064014 5864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:18.071285 5864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:18.071403 5864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:18.071440 5864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:18.071497 5864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:18.071533 5864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 00:53:18.071594 5864 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 00:53:18.071633 5864 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 00:53:18.081617 5864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:18.082921 5864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:18.084204 5864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:18.096426 5869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:18.096325 5872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:18.096397 5870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:18.097565 5864 server_base.cc:1034] running on GCE node
I20260619 00:53:18.098526 5864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:18.100123 5864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:18.101389 5864 hybrid_clock.cc:648] HybridClock initialized: now 1781830398101340 us; error 42 us; skew 500 ppm
I20260619 00:53:18.101804 5864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:53:18.107451 5864 webserver.cc:460] Webserver started at http://127.0.97.62:33879/ using document root <none> and password file <none>
I20260619 00:53:18.108279 5864 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:18.108426 5864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:18.108762 5864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:18.111810 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "461f396aca384e46ba10c6ccae3d6a07"
format_stamp: "Formatted at 2026-06-19 00:53:18 on dist-test-slave-r7bh"
server_key: "2e8aafa3d631e6195a772d7011707e53"
server_key_iv: "ec5e20bf5aec99fd2a0f43bf6614a4fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:18.112962 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "461f396aca384e46ba10c6ccae3d6a07"
format_stamp: "Formatted at 2026-06-19 00:53:18 on dist-test-slave-r7bh"
server_key: "2e8aafa3d631e6195a772d7011707e53"
server_key_iv: "ec5e20bf5aec99fd2a0f43bf6614a4fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:18.118659 5864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:53:18.122360 5878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:18.123898 5864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 00:53:18.124104 5864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "461f396aca384e46ba10c6ccae3d6a07"
format_stamp: "Formatted at 2026-06-19 00:53:18 on dist-test-slave-r7bh"
server_key: "2e8aafa3d631e6195a772d7011707e53"
server_key_iv: "ec5e20bf5aec99fd2a0f43bf6614a4fc"
server_key_version: "encryptionkey@0"
I20260619 00:53:18.124344 5864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:18.146914 5864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:18.147815 5864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:18.148110 5864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:18.166863 5864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34327
I20260619 00:53:18.166903 5929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34327 every 8 connection(s)
I20260619 00:53:18.168298 5864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:18.172267 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:18.177735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5864
I20260619 00:53:18.177934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:18.178730 388 external_mini_cluster.cc:1428] Setting key 04a08589fc1bcc33705d075a3b5a5479
I20260619 00:53:18.183640 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07: Bootstrap starting.
I20260619 00:53:18.187049 5930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:18.188311 5930 log.cc:824] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:18.191809 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07: No bootstrap required, opened a new log
I20260619 00:53:18.193971 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:18.180953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.97.62:34327)
0619 00:53:18.182138 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:18.182151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:18.182198 (+ 47us) server_negotiation.cc:407] Connection header received
0619 00:53:18.184069 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:18.184097 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:18.184600 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:18.185064 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:18.185680 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.187170 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.189063 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.190111 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.190246 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:18.191542 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:18.191551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:18.191564 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:53:18.192192 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:18.192281 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:53:18.192304 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:18.192393 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:18.192642 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 00:53:18.192998 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":465,"threads_started":1}
I20260619 00:53:18.197842 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:53:18.200774 5930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } }
I20260619 00:53:18.201234 5930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:18.201375 5930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461f396aca384e46ba10c6ccae3d6a07, State: Initialized, Role: FOLLOWER
I20260619 00:53:18.201999 5930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } }
I20260619 00:53:18.202246 5930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:18.202396 5930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:18.202579 5930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:18.204694 5930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } }
I20260619 00:53:18.205267 5930 leader_election.cc:302] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461f396aca384e46ba10c6ccae3d6a07; no voters:
I20260619 00:53:18.205936 5930 leader_election.cc:288] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:18.206113 5935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:18.206948 5935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [term 1 LEADER]: Becoming Leader. State: Replica: 461f396aca384e46ba10c6ccae3d6a07, State: Running, Role: LEADER
I20260619 00:53:18.207547 5935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } }
I20260619 00:53:18.208338 5930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:18.210430 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461f396aca384e46ba10c6ccae3d6a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } } }
I20260619 00:53:18.210419 5937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461f396aca384e46ba10c6ccae3d6a07. Latest consensus state: current_term: 1 leader_uuid: "461f396aca384e46ba10c6ccae3d6a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461f396aca384e46ba10c6ccae3d6a07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34327 } } }
I20260619 00:53:18.210831 5937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:18.210831 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:18.211721 5941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:18.217402 5941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:18.226413 5941 catalog_manager.cc:1347] Generated new cluster ID: 854fb390c15f4aaeb3389c49a7eef78e
I20260619 00:53:18.226527 5941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:18.239179 5941 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:18.241070 5941 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:18.248726 5941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461f396aca384e46ba10c6ccae3d6a07: Generated new TSK 0
I20260619 00:53:18.249699 5941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:53:18.370563 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:18.271866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.0.97.62:34327)
0619 00:53:18.272032 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:18.272036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:18.272060 (+ 24us) server_negotiation.cc:407] Connection header received
0619 00:53:18.272347 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:18.272351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:18.272421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:18.272553 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0619 00:53:18.273137 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.273855 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.275844 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.276297 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.276405 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:18.277032 (+ 627us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 00:53:18.369933 (+ 92901us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 00:53:18.370272 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45100: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 00:53:18.370741 5933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45100: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 00:53:18.370911 5959 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:18.271325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34327 (local address 127.0.0.1:45100)
0619 00:53:18.271976 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:18.272005 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:18.272145 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:18.272593 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:18.272599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:18.272611 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 00:53:18.272962 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:18.272977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.274006 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.274011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:18.275675 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:18.275690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:18.276422 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:18.276478 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:18.276665 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:18.276714 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 00:53:18.370209 (+ 93495us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 00:53:18.370502 (+ 293us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":334,"threads_started":1}
I20260619 00:53:18.372012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5864
I20260619 00:53:18.373198 5925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:18.476918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5864
2026-06-19T00:53:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (799 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (411 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1501 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:20 dist-test-slave-r7bh krb5kdc[5975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:20 dist-test-slave-r7bh krb5kdc[5975](info): set up 2 sockets
Jun 19 00:53:20 dist-test-slave-r7bh krb5kdc[5975](info): commencing operation
krb5kdc: starting...
W20260619 00:53:21.477591 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:21 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:21.683943 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37461
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:22.016676 5988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:22.017076 5988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:22.017180 5988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:22.024688 5988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:22.024921 5988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:22.024988 5988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:22.025051 5988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:22.025096 5988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:22.035149 5988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:22.036464 5988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:22.037756 5988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:22.049285 5993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:22.049311 5994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:22.049417 5996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:22.050092 5988 server_base.cc:1034] running on GCE node
I20260619 00:53:22.050971 5988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:22.052603 5988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:22.053835 5988 hybrid_clock.cc:648] HybridClock initialized: now 1781830402053774 us; error 57 us; skew 500 ppm
I20260619 00:53:22.054270 5988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:22 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:22.057474 5988 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:22.058907 5988 webserver.cc:460] Webserver started at http://127.0.97.62:39615/ using document root <none> and password file <none>
I20260619 00:53:22.059731 5988 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:22.059880 5988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:22.060271 5988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:22.064942 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "0564c7f37fc641ec98a97909bc520db3"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "14b8f5e0412c105e71a698d776a7eb7e"
server_key_iv: "0e8e5690a7fa2c351bfbe06b7fdae2c7"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.066114 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "0564c7f37fc641ec98a97909bc520db3"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "14b8f5e0412c105e71a698d776a7eb7e"
server_key_iv: "0e8e5690a7fa2c351bfbe06b7fdae2c7"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.071520 5988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 00:53:22.075131 6003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:22.076582 5988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:22.076794 5988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "0564c7f37fc641ec98a97909bc520db3"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "14b8f5e0412c105e71a698d776a7eb7e"
server_key_iv: "0e8e5690a7fa2c351bfbe06b7fdae2c7"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.077057 5988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:22.093988 5988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:22.098060 5988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:22.098338 5988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:22.116070 5988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37461
I20260619 00:53:22.116098 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37461 every 8 connection(s)
I20260619 00:53:22.117555 5988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:22.121573 6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:22.124099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 5988
I20260619 00:53:22.124322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:22.125113 388 external_mini_cluster.cc:1428] Setting key 3e92dfca6b063a745b8cb2fd5c8dc154
I20260619 00:53:22.134550 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3: Bootstrap starting.
I20260619 00:53:22.138408 6055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:22.139808 6055 log.cc:824] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:53:22 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:22.142890 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3: No bootstrap required, opened a new log
I20260619 00:53:22.148936 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:22.127456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.0.97.62:37461)
0619 00:53:22.128303 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:22.128315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:22.128357 (+ 42us) server_negotiation.cc:407] Connection header received
0619 00:53:22.130220 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:22.130243 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:22.130653 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:22.130994 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:22.132746 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.134065 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.135749 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.136436 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.136567 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:22.141525 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:22.141533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:22.141548 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:22.141580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:22.144653 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:22.145552 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:22.145559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:22.145586 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:22.145752 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:22.146155 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:22.146158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:22.146160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:22.146712 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:22.146946 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:22.147371 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 00:53:22.147760 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":407,"threads_started":1}
I20260619 00:53:22.151690 6055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } }
I20260619 00:53:22.152061 6055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:22.152179 6055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0564c7f37fc641ec98a97909bc520db3, State: Initialized, Role: FOLLOWER
I20260619 00:53:22.152731 6055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } }
I20260619 00:53:22.153017 6055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:22.153162 6055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:22.153322 6055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:22.155655 6055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } }
I20260619 00:53:22.156118 6055 leader_election.cc:302] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0564c7f37fc641ec98a97909bc520db3; no voters:
I20260619 00:53:22.156663 6055 leader_election.cc:288] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:22.156880 6060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:22.157619 6060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [term 1 LEADER]: Becoming Leader. State: Replica: 0564c7f37fc641ec98a97909bc520db3, State: Running, Role: LEADER
I20260619 00:53:22.158183 6060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } }
I20260619 00:53:22.158737 6055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:22.161394 6061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0564c7f37fc641ec98a97909bc520db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } } }
I20260619 00:53:22.161885 6061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:22.161700 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0564c7f37fc641ec98a97909bc520db3. Latest consensus state: current_term: 1 leader_uuid: "0564c7f37fc641ec98a97909bc520db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0564c7f37fc641ec98a97909bc520db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37461 } } }
I20260619 00:53:22.162026 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:22.163142 6069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:22.169642 6069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:22.179713 6069 catalog_manager.cc:1347] Generated new cluster ID: 4a66ed8651a44348959985263843680f
I20260619 00:53:22.179836 6069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:22.221673 6069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:22.223006 6069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:22.236024 6069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0564c7f37fc641ec98a97909bc520db3: Generated new TSK 0
I20260619 00:53:22.236944 6069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:53:22.294539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:22.630028 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:22.630340 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:22.630506 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:22.637823 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:22.637944 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:22.638144 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:22.648984 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:22.650315 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:22.651643 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:22.665393 6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:22.665434 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:22.665462 6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:22.666115 6081 server_base.cc:1034] running on GCE node
I20260619 00:53:22.666730 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:22.667923 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:22.669127 6081 hybrid_clock.cc:648] HybridClock initialized: now 1781830402669057 us; error 63 us; skew 500 ppm
I20260619 00:53:22.669459 6081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:22 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:22.672696 6081 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:53:22.674223 6081 webserver.cc:460] Webserver started at http://127.0.97.1:35389/ using document root <none> and password file <none>
I20260619 00:53:22.674919 6081 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:22.675063 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:22.675364 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:22.678433 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "196f09661eeb4abf8642089d944e82e5"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "de4f2d0bf1a8616e7eb0b18a0ba5044e"
server_key_iv: "44675e77b120c46f0a20a7bdd2a0031f"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.679221 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "196f09661eeb4abf8642089d944e82e5"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "de4f2d0bf1a8616e7eb0b18a0ba5044e"
server_key_iv: "44675e77b120c46f0a20a7bdd2a0031f"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.684746 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:53:22.688433 6096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:22.689884 6081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:22.690089 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "196f09661eeb4abf8642089d944e82e5"
format_stamp: "Formatted at 2026-06-19 00:53:22 on dist-test-slave-r7bh"
server_key: "de4f2d0bf1a8616e7eb0b18a0ba5044e"
server_key_iv: "44675e77b120c46f0a20a7bdd2a0031f"
server_key_version: "encryptionkey@0"
I20260619 00:53:22.690301 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:22.707643 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:22.711943 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:22.712340 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:22.713445 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:22.714875 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:22.714978 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:22.715126 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:22.715194 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:22.748889 6081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36993
I20260619 00:53:22.748945 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36993 every 8 connection(s)
I20260619 00:53:22.750202 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:22.757699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6081
I20260619 00:53:22.757905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:22.758730 388 external_mini_cluster.cc:1428] Setting key f4650721db824b44549a9ba0218f2e64
Jun 19 00:53:22 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:22.773510 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:22.752951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34737 (local address 127.0.97.62:37461)
0619 00:53:22.753401 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:22.753406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:22.754486 (+ 1080us) server_negotiation.cc:407] Connection header received
0619 00:53:22.756269 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:22.756275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:22.756364 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:22.756480 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:22.758551 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.759401 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.761312 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.761707 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.761840 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:22.765773 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:22.765781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:22.765787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:22.765842 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:22.769078 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:22.770172 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:22.770180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:22.770184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:22.770354 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:22.770992 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:22.770997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:22.771000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:22.771367 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:22.771519 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:22.772702 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0619 00:53:22.773098 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260619 00:53:22.774452 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:22.753587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37461 (local address 127.0.97.1:34737)
0619 00:53:22.754317 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:22.754356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:22.755991 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:22.756739 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:22.756748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:22.757465 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:22.758353 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:22.758381 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.759583 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.759589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:22.761163 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:22.761175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:22.761934 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:22.761942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:22.762196 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:22.762945 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:22.762973 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:22.765499 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:22.769240 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:22.769250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:22.769271 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:22.769995 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:22.770522 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:22.770526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:22.770530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:22.770848 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:22.771600 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:22.771610 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:22.772101 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 00:53:22.772452 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 00:53:22.773100 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260619 00:53:22.776165 6209 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37461
I20260619 00:53:22.776515 6209 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:22.777520 6209 heartbeater.cc:511] Master 127.0.97.62:37461 requested a full tablet report, sending...
I20260619 00:53:22.780045 6020 ts_manager.cc:194] Registered new tserver with Master: 196f09661eeb4abf8642089d944e82e5 (127.0.97.1:36993)
I20260619 00:53:22.782055 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34737
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:53:22.811812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:23.146524 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:23.146826 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:23.146981 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:23.154323 6215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:23.154436 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:23.154605 6215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:23.164532 6215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:23.165877 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:23.167173 6215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:23.180048 6220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:23.180058 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:23.180127 6223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:23.181356 6215 server_base.cc:1034] running on GCE node
I20260619 00:53:23.182019 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:23.182914 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:23.184108 6215 hybrid_clock.cc:648] HybridClock initialized: now 1781830403184049 us; error 47 us; skew 500 ppm
I20260619 00:53:23.184425 6215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.190537 6215 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:53:23.191953 6215 webserver.cc:460] Webserver started at http://127.0.97.2:37759/ using document root <none> and password file <none>
I20260619 00:53:23.192620 6215 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:23.192744 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:23.193099 6215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:23.196184 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "ab4eea483d8f4941993dcf8db910b4b8"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "7c45ee5b15013f9678d3159533c16ee1"
server_key_iv: "49b0ce2215da690a0fe947200c6b600e"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.197007 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ab4eea483d8f4941993dcf8db910b4b8"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "7c45ee5b15013f9678d3159533c16ee1"
server_key_iv: "49b0ce2215da690a0fe947200c6b600e"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.202226 6215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:23.205755 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.207165 6215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:23.207345 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "ab4eea483d8f4941993dcf8db910b4b8"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "7c45ee5b15013f9678d3159533c16ee1"
server_key_iv: "49b0ce2215da690a0fe947200c6b600e"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.207554 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:23.230412 6215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:23.234440 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:23.234732 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:23.235736 6215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:23.237089 6215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:23.237169 6215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.237305 6215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:23.237357 6215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.270118 6215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33313
I20260619 00:53:23.270210 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33313 every 8 connection(s)
I20260619 00:53:23.271369 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:23.272987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6215
I20260619 00:53:23.273167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:23.273728 388 external_mini_cluster.cc:1428] Setting key 566fc4713f2b15bc52f93fbf19eb44cb
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:23.295300 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.274022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53217 (local address 127.0.97.62:37461)
0619 00:53:23.274537 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.274543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.275654 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 00:53:23.278449 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.278455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.278594 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.278762 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.280903 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.281782 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.283634 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.284043 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.284140 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.288190 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.288199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.288206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.288250 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.291887 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.292606 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.292615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.292622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.292797 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.293249 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.293257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.293263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.293661 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.293819 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.294599 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.294847 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":310,"threads_started":1}
I20260619 00:53:23.296005 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.274641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37461 (local address 127.0.97.2:53217)
0619 00:53:23.275526 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:23.275574 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:23.278127 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:23.279064 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:23.279079 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:23.279867 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:23.280695 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:23.280722 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.281941 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.281949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:23.283519 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:23.283530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.284276 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.284286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:23.284624 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.285355 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:23.285388 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:23.287970 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:23.292026 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:23.292032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:23.292047 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:23.292463 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:23.292932 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:23.292935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:23.292938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:23.293115 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:23.293784 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:23.293790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:23.294168 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 00:53:23.294439 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 00:53:23.294843 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:53:23.297622 6343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37461
I20260619 00:53:23.297972 6343 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:23.298735 6343 heartbeater.cc:511] Master 127.0.97.62:37461 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:53:23.300469 6020 ts_manager.cc:194] Registered new tserver with Master: ab4eea483d8f4941993dcf8db910b4b8 (127.0.97.2:33313)
I20260619 00:53:23.301787 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:53:23.324162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:23.650790 6349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:23.651103 6349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:23.651268 6349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:23.658465 6349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:23.658643 6349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:23.658823 6349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:23.668880 6349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:23.670236 6349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:23.671590 6349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:23.685238 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:23.685164 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:23.685195 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:23.686199 6349 server_base.cc:1034] running on GCE node
I20260619 00:53:23.686818 6349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:23.687794 6349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:23.689046 6349 hybrid_clock.cc:648] HybridClock initialized: now 1781830403688969 us; error 50 us; skew 500 ppm
I20260619 00:53:23.689384 6349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.693025 6349 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:53:23.694583 6349 webserver.cc:460] Webserver started at http://127.0.97.3:34735/ using document root <none> and password file <none>
I20260619 00:53:23.695304 6349 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:23.695432 6349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:23.695744 6349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:23.698814 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "a074fcc95ab0431c927b7ea544f3ad1f"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "0136c10ada154fb8ba5a5e8f60bff141"
server_key_iv: "c8ee13eb50bffa9764ea34b0a73d3060"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.699756 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a074fcc95ab0431c927b7ea544f3ad1f"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "0136c10ada154fb8ba5a5e8f60bff141"
server_key_iv: "c8ee13eb50bffa9764ea34b0a73d3060"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.705678 6349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 00:53:23.710363 6364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.712234 6349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 00:53:23.712425 6349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "a074fcc95ab0431c927b7ea544f3ad1f"
format_stamp: "Formatted at 2026-06-19 00:53:23 on dist-test-slave-r7bh"
server_key: "0136c10ada154fb8ba5a5e8f60bff141"
server_key_iv: "c8ee13eb50bffa9764ea34b0a73d3060"
server_key_version: "encryptionkey@0"
I20260619 00:53:23.712685 6349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:23.760795 6349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:23.765419 6349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:23.765717 6349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:23.766744 6349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:23.768118 6349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:23.768191 6349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.768316 6349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:23.768384 6349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:23.786159 6209 heartbeater.cc:503] Master 127.0.97.62:37461 was elected leader, sending a full tablet report...
I20260619 00:53:23.801229 6349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37433
I20260619 00:53:23.801265 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37433 every 8 connection(s)
I20260619 00:53:23.802543 6349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:23.804412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6349
I20260619 00:53:23.804586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:23.805125 388 external_mini_cluster.cc:1428] Setting key 2b1ceb20f03f6592907074a54a95db6b
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:23.823009 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.805256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34693 (local address 127.0.97.62:37461)
0619 00:53:23.805769 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.805773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.807021 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 00:53:23.808942 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.808947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.809031 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.809147 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.810606 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.811448 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.813053 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.813439 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.813566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.816980 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.816986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.816991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.817028 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.819753 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.820621 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.820627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.820631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.820760 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.821238 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.821244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.821247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.821523 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.821669 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.822371 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.822598 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":239,"threads_started":1}
I20260619 00:53:23.823975 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.805920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37461 (local address 127.0.97.3:34693)
0619 00:53:23.806854 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:23.806914 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:23.808644 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:23.809392 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:23.809401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:23.809894 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:23.810439 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:23.810455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.811598 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.811603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:23.812920 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:23.812931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.813656 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.813664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:23.813903 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.814538 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:23.814559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:23.816740 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:23.819906 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:23.819915 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:23.819933 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:23.820499 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:23.820917 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:23.820922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:23.820925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:23.821121 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:23.821656 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:23.821664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:23.822018 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 00:53:23.822289 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 00:53:23.822845 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":361,"threads_started":1}
I20260619 00:53:23.825457 6477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37461
I20260619 00:53:23.825778 6477 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:23.826467 6477 heartbeater.cc:511] Master 127.0.97.62:37461 requested a full tablet report, sending...
I20260619 00:53:23.827934 6020 ts_manager.cc:194] Registered new tserver with Master: a074fcc95ab0431c927b7ea544f3ad1f (127.0.97.3:37433)
I20260619 00:53:23.828688 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34693
I20260619 00:53:23.832643 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/test-admin.keytab.
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.858110 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 00:53:23.863667 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:23.860785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.0.97.1:36993)
0619 00:53:23.861601 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.861612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.861650 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:53:23.861730 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.861737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.862044 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.862216 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.863214 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57308: BlockingRecv error: recv got EOF from 127.0.0.1:57308 (error 108)
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":360,"threads_started":1}
W20260619 00:53:23.863628 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.865271 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.866557 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:23.882920 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.868417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.1:36993)
0619 00:53:23.868631 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.868639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.868860 (+ 221us) server_negotiation.cc:407] Connection header received
0619 00:53:23.869073 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.869078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.869144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.869249 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.870208 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.871191 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.872375 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.873013 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.873141 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.876658 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.876681 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.876702 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.876743 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.880004 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.880779 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.880792 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.880794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.880979 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.881355 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.881360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.881363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.881773 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.882055 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.882372 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.882618 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 00:53:23.888535 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:23.886932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.0.97.1:36993)
0619 00:53:23.887120 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.887127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.887268 (+ 141us) server_negotiation.cc:407] Connection header received
0619 00:53:23.887448 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.887453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.887528 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.887622 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.888302 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57326: BlockingRecv error: recv got EOF from 127.0.0.1:57326 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 00:53:23.888725 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.890224 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.891507 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:23.904996 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.892764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.0.97.1:36993)
0619 00:53:23.893019 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.893025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.893179 (+ 154us) server_negotiation.cc:407] Connection header received
0619 00:53:23.893293 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.893296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.893349 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.893428 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.894362 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.895176 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.896288 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.896704 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.896919 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.900109 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.900114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.900117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.900159 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.902687 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.903280 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.903284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.903286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.903403 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.903728 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.903731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.903733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.903967 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.904149 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.904426 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.904652 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260619 00:53:23.909284 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:23.907488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.0.97.1:36993)
0619 00:53:23.907685 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.907693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.907976 (+ 283us) server_negotiation.cc:407] Connection header received
0619 00:53:23.908149 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.908154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.908234 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.908328 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.909014 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57346: BlockingRecv error: recv got EOF from 127.0.0.1:57346 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 00:53:23.909479 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830351410501-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.911128 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:23.912551 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 00:53:23 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:23.926982 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.913932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.97.1:36993)
0619 00:53:23.914213 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.914221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.914435 (+ 214us) server_negotiation.cc:407] Connection header received
0619 00:53:23.914635 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.914640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.914704 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.914804 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.915743 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.916532 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.917813 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.918235 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.918363 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.921908 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.921912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.921916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.921948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.924301 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.925076 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.925081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.925085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.925262 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.925632 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.925636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.925638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.925892 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.926211 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.926466 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.926683 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:53:23.928530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6081
I20260619 00:53:23.941166 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:23.929277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.0.97.1:36993)
0619 00:53:23.929417 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:23.929421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:23.929778 (+ 357us) server_negotiation.cc:407] Connection header received
0619 00:53:23.929894 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:23.929898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:23.929951 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:23.930028 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:23.930912 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.931670 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.933074 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:23.933485 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:23.933578 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:23.935725 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:23.935730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:23.935734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:23.935767 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:23.938571 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.939258 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.939262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.939265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.939413 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:23.939835 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:23.939839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:23.939842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:23.940108 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:23.940298 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:23.940629 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 00:53:23.940892 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 00:53:23.941741 6204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:24.025475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6081
I20260619 00:53:24.042439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6215
Jun 19 00:53:24 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:53:24.058035 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:24.043389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.0.97.2:33313)
0619 00:53:24.043941 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:24.043949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:24.043975 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:53:24.044036 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:24.044042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:24.044256 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:24.044385 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:24.045430 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:24.046463 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:24.047804 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:24.048384 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:24.048499 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:24.052429 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:24.052442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:24.052461 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:24.052506 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:24.055394 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:24.056092 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:24.056099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:24.056101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:24.056230 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:24.056546 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:24.056549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:24.056551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:24.057053 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:24.057283 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:24.057539 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 00:53:24.057751 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260619 00:53:24.059434 6338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:24.142694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6215
I20260619 00:53:24.158557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6349
Jun 19 00:53:24 dist-test-slave-r7bh krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:53:24.174799 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:24.159743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.97.3:37433)
0619 00:53:24.160439 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:24.160446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:24.160475 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:53:24.160535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:24.160550 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:24.160767 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:24.160953 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:24.161914 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:24.163024 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:24.164557 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:24.165318 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:24.165441 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:24.169354 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:24.169366 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:24.169380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:24.169413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:24.172207 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:24.173036 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:24.173043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:24.173045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:24.173208 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:24.173511 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:24.173513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:24.173515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:24.173823 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:24.174048 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:24.174254 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 00:53:24.174497 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":349,"threads_started":1}
I20260619 00:53:24.176059 6472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:24.262319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6349
I20260619 00:53:24.277974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 5988
I20260619 00:53:24.279129 6050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:24.382259 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 5988
2026-06-19T00:53:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3998 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:24 dist-test-slave-r7bh krb5kdc[6505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:24 dist-test-slave-r7bh krb5kdc[6505](info): set up 2 sockets
Jun 19 00:53:24 dist-test-slave-r7bh krb5kdc[6505](info): commencing operation
krb5kdc: starting...
W20260619 00:53:25.481743 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:25 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 00:53:25.692174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:26.018198 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:26.018513 6518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:26.018675 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:26.026120 6518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:26.026240 6518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:26.026320 6518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:26.026422 6518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:26.026484 6518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:26.036363 6518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:26.037817 6518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:26.039227 6518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:26.050833 6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:26.050992 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:26.051366 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:26.052058 6518 server_base.cc:1034] running on GCE node
I20260619 00:53:26.052975 6518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:26.054427 6518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:26.055686 6518 hybrid_clock.cc:648] HybridClock initialized: now 1781830406055629 us; error 40 us; skew 500 ppm
I20260619 00:53:26.056094 6518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:26 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830406, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:26.059284 6518 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 00:53:26.060884 6518 webserver.cc:460] Webserver started at http://127.0.97.62:37153/ using document root <none> and password file <none>
I20260619 00:53:26.061585 6518 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:26.061721 6518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:26.062037 6518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:26.065030 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "f3e506e1c3364b11af306c7f83c7186b"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "e937bcc1637feb0ebf62ebc8aee56007"
server_key_iv: "bca4e047f668b51021a395d475722205"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.065793 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3e506e1c3364b11af306c7f83c7186b"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "e937bcc1637feb0ebf62ebc8aee56007"
server_key_iv: "bca4e047f668b51021a395d475722205"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.071183 6518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:53:26.075002 6533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:26.076488 6518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:26.076695 6518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "f3e506e1c3364b11af306c7f83c7186b"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "e937bcc1637feb0ebf62ebc8aee56007"
server_key_iv: "bca4e047f668b51021a395d475722205"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.076944 6518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:26.100153 6518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:26.104916 6518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:26.105238 6518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:26.124343 6518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41499
I20260619 00:53:26.124361 6584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41499 every 8 connection(s)
I20260619 00:53:26.125974 6518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:26.129848 6585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:26.132683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6518
I20260619 00:53:26.132911 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:26.133701 388 external_mini_cluster.cc:1428] Setting key c31d96eb4955c1249548c1e284cf4a2d
I20260619 00:53:26.142364 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Bootstrap starting.
I20260619 00:53:26.146144 6585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:26.147475 6585 log.cc:824] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:53:26 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:26.150775 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: No bootstrap required, opened a new log
I20260619 00:53:26.157652 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:26.135847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.0.97.62:41499)
0619 00:53:26.136645 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:26.136658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:26.136703 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:53:26.138428 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:26.138451 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:26.138869 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:26.139198 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:26.140911 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.142200 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.143930 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.144602 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.144755 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:26.149856 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:26.149863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:26.149878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:26.149918 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:26.153156 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:26.154024 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:26.154030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:26.154037 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:26.154226 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:26.154683 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:26.154687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:26.154693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:26.155272 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:26.155515 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:26.156007 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 00:53:26.156444 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":401,"threads_started":1}
I20260619 00:53:26.159804 6585 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } }
I20260619 00:53:26.160281 6585 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:26.160385 6585 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e506e1c3364b11af306c7f83c7186b, State: Initialized, Role: FOLLOWER
I20260619 00:53:26.161046 6585 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } }
I20260619 00:53:26.161278 6585 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:26.161428 6585 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:26.161592 6585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:26.163780 6585 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } }
I20260619 00:53:26.164335 6585 leader_election.cc:302] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e506e1c3364b11af306c7f83c7186b; no voters:
I20260619 00:53:26.165050 6585 leader_election.cc:288] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:26.165159 6590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:26.165920 6590 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 LEADER]: Becoming Leader. State: Replica: f3e506e1c3364b11af306c7f83c7186b, State: Running, Role: LEADER
I20260619 00:53:26.166479 6590 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } }
I20260619 00:53:26.167328 6585 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:26.169394 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } }
I20260619 00:53:26.169458 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e506e1c3364b11af306c7f83c7186b. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } }
I20260619 00:53:26.169778 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:26.169776 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:26.175010 6596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:26.180346 6596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:26.189378 6596 catalog_manager.cc:1347] Generated new cluster ID: 27d07f98f5a44951a0748fd462b1060e
I20260619 00:53:26.189517 6596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:26.202117 6596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:26.203524 6596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:26.216845 6596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Generated new TSK 0
I20260619 00:53:26.217759 6596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 00:53:26.278546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:26.651258 6611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:26.651526 6611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:26.651618 6611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:26.658732 6611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:26.658836 6611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:26.658951 6611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:26.668720 6611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:26.670012 6611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:26.671248 6611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:26.684159 6617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:26.684228 6616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:26.684201 6619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:26.685114 6611 server_base.cc:1034] running on GCE node
I20260619 00:53:26.685680 6611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:26.686625 6611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:26.687822 6611 hybrid_clock.cc:648] HybridClock initialized: now 1781830406687771 us; error 43 us; skew 500 ppm
I20260619 00:53:26.688150 6611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:26 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830406, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:26.691455 6611 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 00:53:26.693153 6611 webserver.cc:460] Webserver started at http://127.0.97.1:36623/ using document root <none> and password file <none>
I20260619 00:53:26.693858 6611 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:26.693998 6611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:26.694300 6611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:26.697365 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "7df5f21f0a558889c233c3df889e4ca6"
server_key_iv: "f4d44ed74b7b673b53f3b78e048a2efd"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.698172 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "7df5f21f0a558889c233c3df889e4ca6"
server_key_iv: "f4d44ed74b7b673b53f3b78e048a2efd"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.703644 6611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:26.707216 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:26.708628 6611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:53:26.708803 6611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "7df5f21f0a558889c233c3df889e4ca6"
server_key_iv: "f4d44ed74b7b673b53f3b78e048a2efd"
server_key_version: "encryptionkey@0"
I20260619 00:53:26.709065 6611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:26.736312 6611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:26.740499 6611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:26.740794 6611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:26.741818 6611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:26.743217 6611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:26.743314 6611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:26.743453 6611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:26.743520 6611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:26.778836 6611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36365
I20260619 00:53:26.778878 6738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36365 every 8 connection(s)
I20260619 00:53:26.780194 6611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:26.783918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6611
I20260619 00:53:26.784137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:26.784976 388 external_mini_cluster.cc:1428] Setting key 57dfd835207fa2a3e819e9f5a2b4668c
Jun 19 00:53:26 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830406, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:26.800393 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:26.782880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34841 (local address 127.0.97.62:41499)
0619 00:53:26.783320 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:26.783324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:26.784400 (+ 1076us) server_negotiation.cc:407] Connection header received
0619 00:53:26.786471 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:26.786476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:26.786564 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:26.786674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:26.788169 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.788997 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.790514 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.790901 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.791029 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:26.794832 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:26.794837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:26.794841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:26.794873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:26.797405 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:26.798110 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:26.798117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:26.798121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:26.798271 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:26.798686 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:26.798694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:26.798697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:26.798954 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:26.799106 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:26.799794 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 00:53:26.800011 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":241,"threads_started":1}
I20260619 00:53:26.801501 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:26.783348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:34841)
0619 00:53:26.784244 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:26.784294 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:26.786213 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:26.786911 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:26.786919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:26.787439 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:26.788005 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:26.788023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.789142 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.789146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:26.790359 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:26.790370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:26.791130 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:26.791136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:26.791374 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:26.792070 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:26.792113 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:26.794620 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:26.797553 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:26.797562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:26.797578 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:26.797992 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:26.798377 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:26.798382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:26.798386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:26.798577 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:26.799083 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:26.799091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:26.799417 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 00:53:26.799715 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 00:53:26.800247 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":327,"threads_started":1}
I20260619 00:53:26.803263 6739 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:26.803568 6739 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:53:26.804308 6739 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260619 00:53:26.806608 6550 ts_manager.cc:194] Registered new tserver with Master: bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365)
I20260619 00:53:26.808069 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 00:53:26.830040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:27.162602 6745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:27.162879 6745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:27.163065 6745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:27.170392 6745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:27.170506 6745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:27.170684 6745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:27.180583 6745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:27.181969 6745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:27.183332 6745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:27.196237 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:27.196237 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:27.196440 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:27.196708 6745 server_base.cc:1034] running on GCE node
I20260619 00:53:27.197583 6745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:27.198510 6745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:27.199726 6745 hybrid_clock.cc:648] HybridClock initialized: now 1781830407199677 us; error 38 us; skew 500 ppm
I20260619 00:53:27.200035 6745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:27 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830407, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:27.203225 6745 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 00:53:27.204578 6745 webserver.cc:460] Webserver started at http://127.0.97.2:41547/ using document root <none> and password file <none>
I20260619 00:53:27.205294 6745 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:27.205415 6745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:27.205708 6745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:27.208492 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "6807ccc12ab74177a2774fc89f23df1b"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "381fa2483f144a6cfa72aaa42bbfbd46"
server_key_iv: "253a99133d68d5abf7aa6e71b52368d1"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.209350 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6807ccc12ab74177a2774fc89f23df1b"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "381fa2483f144a6cfa72aaa42bbfbd46"
server_key_iv: "253a99133d68d5abf7aa6e71b52368d1"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.214499 6745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:27.218009 6760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.219352 6745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:27.219508 6745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "6807ccc12ab74177a2774fc89f23df1b"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "381fa2483f144a6cfa72aaa42bbfbd46"
server_key_iv: "253a99133d68d5abf7aa6e71b52368d1"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.219702 6745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:27.239816 6745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:27.243968 6745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:27.244270 6745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:27.245234 6745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:27.246558 6745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:27.246628 6745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.246791 6745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:27.246841 6745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.278925 6745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40347
I20260619 00:53:27.278963 6872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40347 every 8 connection(s)
I20260619 00:53:27.280277 6745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:27.290266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6745
I20260619 00:53:27.290436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:27.290939 388 external_mini_cluster.cc:1428] Setting key 12358862153e6046d058808e0195976c
Jun 19 00:53:27 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830407, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:27.301075 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:27.282916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54021 (local address 127.0.97.62:41499)
0619 00:53:27.283111 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:27.283116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:27.284242 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 00:53:27.286079 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:27.286085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:27.286170 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:27.286283 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:27.287733 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.288432 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.290091 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.290653 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.290759 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:27.295400 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:27.295411 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:27.295415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:27.295447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:27.298047 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:27.298791 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:27.298798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:27.298801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:27.298925 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:27.299356 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:27.299361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:27.299364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:27.299633 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:27.299777 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:27.300479 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 00:53:27.300710 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 00:53:27.302227 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:27.283364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:54021)
0619 00:53:27.284081 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:27.284120 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:27.285815 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:27.286511 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:27.286519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:27.287007 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:27.287566 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:27.287584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.288571 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.288576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:27.289873 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:27.289892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.290898 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.290906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:27.291234 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:27.292091 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:27.292145 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:27.295170 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:27.298214 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:27.298223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:27.298241 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:27.298665 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:27.299031 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:27.299038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:27.299041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:27.299243 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:27.299754 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:27.299762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:27.300121 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 00:53:27.300381 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 00:53:27.300972 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
I20260619 00:53:27.303850 6873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:27.304170 6873 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:27.305011 6873 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
I20260619 00:53:27.306537 6550 ts_manager.cc:194] Registered new tserver with Master: 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:27.307840 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54021
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 00:53:27.339839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:27.660564 6878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:27.660907 6878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:27.661083 6878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:27.668268 6878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:27.668406 6878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:27.668572 6878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:27.678370 6878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:27.679700 6878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:27.681128 6878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:27.694468 6884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:27.694548 6886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:27.694473 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:27.695268 6878 server_base.cc:1034] running on GCE node
I20260619 00:53:27.695813 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:27.697043 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:27.698275 6878 hybrid_clock.cc:648] HybridClock initialized: now 1781830407698217 us; error 49 us; skew 500 ppm
I20260619 00:53:27.698601 6878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:27 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830407, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:27.701635 6878 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 00:53:27.703168 6878 webserver.cc:460] Webserver started at http://127.0.97.3:39783/ using document root <none> and password file <none>
I20260619 00:53:27.703881 6878 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:27.703984 6878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:27.704242 6878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:27.707453 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "da1c9abb05d7fe5f8d4337ccfdfa53c1"
server_key_iv: "c4b521f18c8fac598553caa4e1d804a9"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.708215 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "da1c9abb05d7fe5f8d4337ccfdfa53c1"
server_key_iv: "c4b521f18c8fac598553caa4e1d804a9"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.713682 6878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 00:53:27.717232 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.718547 6878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:53:27.718725 6878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "da1c9abb05d7fe5f8d4337ccfdfa53c1"
server_key_iv: "c4b521f18c8fac598553caa4e1d804a9"
server_key_version: "encryptionkey@0"
I20260619 00:53:27.718906 6878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:27.741137 6878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:27.745339 6878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:27.745627 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:27.746620 6878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:27.748005 6878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:27.748082 6878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.748248 6878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:27.748299 6878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:27.780413 6878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43991
I20260619 00:53:27.780443 7005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43991 every 8 connection(s)
I20260619 00:53:27.781735 6878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:27.790228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 6878
I20260619 00:53:27.790419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:27.790958 388 external_mini_cluster.cc:1428] Setting key f036b0912ffdd475a7691de6d7d079eb
Jun 19 00:53:27 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830407, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:27.801055 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:27.784323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55077 (local address 127.0.97.62:41499)
0619 00:53:27.784491 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:27.784496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:27.785659 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 00:53:27.787323 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:27.787328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:27.787391 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:27.787476 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:27.788802 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.789601 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.791826 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.792357 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.792478 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:27.795933 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:27.795938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:27.795941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:27.795971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:27.798234 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:27.798901 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:27.798908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:27.798911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:27.799033 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:27.799425 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:27.799431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:27.799435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:27.799680 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:27.799831 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:27.800486 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 00:53:27.800688 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:53:27.801952 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:27.784786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:55077)
0619 00:53:27.785516 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:27.785551 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:27.787089 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:27.787657 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:27.787665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:27.788117 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:27.788661 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:27.788678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.789756 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.789760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:27.791677 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:27.791689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:27.792570 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:27.792578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:27.792914 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:27.793591 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:27.793634 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:27.795737 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:27.798364 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:27.798373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:27.798388 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:27.798789 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:27.799135 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:27.799141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:27.799144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:27.799322 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:27.799805 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:27.799813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:27.800130 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 00:53:27.800393 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 00:53:27.800916 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":262,"threads_started":1}
I20260619 00:53:27.803228 7006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:27.803519 7006 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:27.804262 7006 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
I20260619 00:53:27.805711 6550 ts_manager.cc:194] Registered new tserver with Master: 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:27.806151 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:53:27.806522 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55077
I20260619 00:53:27.811703 6739 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 00:53:27.852366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39429
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41499,127.0.97.61:39429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:28.185750 7011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:28.186123 7011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:28.186278 7011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:28.193702 7011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:28.193841 7011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:28.193933 7011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:28.194020 7011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:28.194108 7011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:28.203876 7011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41499,127.0.97.61:39429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
W20260619 00:53:28.204264 7011 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:41499,127.0.97.61:39429'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 00:53:28.205386 7011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:28.206691 7011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:28.218461 7017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:28.218571 7016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:28.218546 7019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:28.219506 7011 server_base.cc:1034] running on GCE node
I20260619 00:53:28.220332 7011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:28.221849 7011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:28.223089 7011 hybrid_clock.cc:648] HybridClock initialized: now 1781830408223027 us; error 42 us; skew 500 ppm
I20260619 00:53:28.223555 7011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:28.230149 7011 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 00:53:28.231752 7011 webserver.cc:460] Webserver started at http://127.0.97.61:34315/ using document root <none> and password file <none>
I20260619 00:53:28.232463 7011 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:28.232587 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:28.232945 7011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:28.235914 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/instance:
uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "5e34565f245edad5c88c95b358428186"
server_key_iv: "707f9329a9c74babafdbd372341a8025"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.236652 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal/instance:
uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "5e34565f245edad5c88c95b358428186"
server_key_iv: "707f9329a9c74babafdbd372341a8025"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.241607 7011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:28.245103 7026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:28.246419 7011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:28.246595 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "5e34565f245edad5c88c95b358428186"
server_key_iv: "707f9329a9c74babafdbd372341a8025"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.246805 7011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:28.267380 7011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:28.271780 7011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:28.272008 7011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:28.290728 7011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39429
I20260619 00:53:28.290776 7077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39429 every 8 connection(s)
I20260619 00:53:28.292104 7011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
I20260619 00:53:28.296450 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:28.302538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7011
I20260619 00:53:28.303658 7078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:53:28.310599 6873 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:28.323994 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.306616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.97.62:41499)
0619 00:53:28.307083 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.307090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.307553 (+ 463us) server_negotiation.cc:407] Connection header received
0619 00:53:28.308300 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.308306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.308411 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.308521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.309787 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.310521 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.312687 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.313271 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.313408 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.317405 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.317410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.317413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.317442 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.320264 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.321260 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.321263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.321265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.321374 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.321801 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.321806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.321809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.322224 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.322335 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.323327 (+ 992us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.323554 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
I20260619 00:53:28.325179 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.306594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33548)
0619 00:53:28.307413 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.307448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.308014 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.308602 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.308616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.309271 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.309622 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.309640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.310652 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.310658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.312551 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.312565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.313478 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.313482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.313729 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.314440 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.314471 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.317210 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.320386 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.320396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.320408 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.321148 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.321475 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.321483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.321488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.321669 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.322333 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.322341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.322848 (+ 507us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.323194 (+ 346us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.323679 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":434,"threads_started":1}
I20260619 00:53:28.325198 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.304914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.0.97.61:39429)
0619 00:53:28.305664 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.305674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.305715 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:53:28.308002 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.308022 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.308255 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.308410 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.309807 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.311231 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.312975 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.313850 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.313958 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.318324 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.318331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.318344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.318370 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.321408 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.322178 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.322184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.322186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.322333 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.322773 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.322787 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.322791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.323248 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.323514 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.323955 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.324174 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"spinlock_wait_cycles":22272,"thread_start_us":353,"threads_started":1}
I20260619 00:53:28.327251 7078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:28.339675 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.328202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.0.97.61:39429)
0619 00:53:28.328376 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.328382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.328585 (+ 203us) server_negotiation.cc:407] Connection header received
0619 00:53:28.328760 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.328765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.328870 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.328961 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.329562 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.330253 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.331352 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.331723 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.331846 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.334824 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.334829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.334832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.334859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.337180 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.337745 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.337749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.337751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.337863 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.338313 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.338317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.338319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.338628 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.338748 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.339097 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.339408 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":58}
I20260619 00:53:28.339690 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.328105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51528)
0619 00:53:28.328507 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.328521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.328637 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.328975 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.328978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.329328 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.329442 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.329450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.330369 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.330376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.331223 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.331233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.331853 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.331859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.332071 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.332574 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.332590 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.334667 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.337298 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.337301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.337305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.337651 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.337999 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.338003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.338006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.338182 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.338751 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.338755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.338893 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.339019 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.339283 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":33}
I20260619 00:53:28.344858 7078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Bootstrap starting.
I20260619 00:53:28.347954 7078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:28.349056 7078 log.cc:824] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:28.352133 7078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: No bootstrap required, opened a new log
I20260619 00:53:28.359319 7078 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } }
I20260619 00:53:28.359679 7078 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:28.359784 7078 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6d1e41c6a74837add03c2f4ae2b8ae, State: Initialized, Role: FOLLOWER
I20260619 00:53:28.360392 7078 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } }
I20260619 00:53:28.361387 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } }
I20260619 00:53:28.361690 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:28.362355 7078 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:28.374884 7011 master_runner.cc:428] Detected that this master cd6d1e41c6a74837add03c2f4ae2b8ae is joining an existing cluster
W20260619 00:53:28.374897 7096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 00:53:28.375059 7011 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:39429
W20260619 00:53:28.375097 7096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 00:53:28.406570 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.379073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33558)
0619 00:53:28.388128 (+ 9055us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.388157 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.388726 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.389397 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.389404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.390145 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.390669 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.390702 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.392287 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.392296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.394702 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.394722 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.396997 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.397005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.397252 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.398027 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.398052 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.399473 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.403581 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.403586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.403593 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.403981 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.404431 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.404439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.404443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.404594 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.405254 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.405263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.405502 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.405691 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.406034 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8835,"thread_start_us":15903,"threads_started":1}
I20260619 00:53:28.408352 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.379103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.97.62:41499)
0619 00:53:28.379315 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.379323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.388278 (+ 8955us) server_negotiation.cc:407] Connection header received
0619 00:53:28.388927 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.388936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.389064 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.389142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.390864 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.392078 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.395706 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.396510 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.396652 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.399752 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.399759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.399763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.399809 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.403446 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.404178 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.404184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.404188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.404321 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.404711 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.404717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.404720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.405115 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.405259 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.406601 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.408139 (+ 1538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260619 00:53:28.410859 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.379416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.97.61:39429)
0619 00:53:28.379564 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.379568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.395479 (+ 15911us) server_negotiation.cc:407] Connection header received
0619 00:53:28.395651 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.395657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.395735 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.395862 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.397315 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.398827 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.401166 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.402116 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.402248 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.404852 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.404858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.404863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.404895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.407937 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.408730 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.408735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.408739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.408890 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.409298 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.409302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.409305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.409641 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.409799 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.410267 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.410494 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 00:53:28.413028 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.379407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51532)
0619 00:53:28.395352 (+ 15945us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.395377 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.395520 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.396029 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.396036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.396774 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.397159 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.397171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.399055 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.399060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.401006 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.401026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.402544 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.402550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.402735 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.403389 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.403415 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.404603 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.408060 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.408066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.408069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.408605 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.409015 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.409019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.409022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.409190 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.409864 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.409868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.410001 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.411089 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.411404 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15755,"thread_start_us":8761,"threads_started":1}
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830406, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:28.435026 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.417025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.0.97.61:39429)
0619 00:53:28.417217 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.417225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.417290 (+ 65us) server_negotiation.cc:407] Connection header received
0619 00:53:28.417682 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.417690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.417796 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.417875 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.419488 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.420707 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.422636 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.423522 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.423670 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.428019 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.428028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.428034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.428072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.431972 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.432725 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.432733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.432737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.432950 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.433353 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.433358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.433362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.433691 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.433847 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.434345 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.434624 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:53:28.435014 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.416480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51546)
0619 00:53:28.417163 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.417211 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.417428 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.418173 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.418181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.419020 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.419317 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.419330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.420889 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.420895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.422496 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.422511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.423691 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.423698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.423905 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.424669 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.424693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.427786 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.432102 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.432111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.432117 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.432594 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.433061 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.433066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.433070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.433241 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.433819 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.433828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.434028 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.434228 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.434545 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":751,"threads_started":1}
I20260619 00:53:28.437181 6550 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:39429: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 39429
}
attrs {
promote: true
}
}
dest_uuid: "f3e506e1c3364b11af306c7f83c7186b"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 00:53:28.438131 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } }
I20260619 00:53:28.439790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33489
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:53:28.454521 7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:28.460093 7053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 00:53:28.462378 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:28.461946 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.466929 7111 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:39429
I20260619 00:53:28.467514 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.467643 7011 master.cc:537] Master@127.0.97.61:39429 shutting down...
I20260619 00:53:28.467715 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:28.494863 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.505151 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.506150 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:28.505699 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.506567 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:28.506577 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e506e1c3364b11af306c7f83c7186b. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.506929 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:28.508805 7011 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Raft consensus shutting down.
I20260619 00:53:28.509337 7011 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Raft consensus is shut down!
I20260619 00:53:28.509527 7011 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: stopping tablet replica
I20260619 00:53:28.529122 7011 master.cc:552] Master@127.0.97.61:39429 shutdown complete.
I20260619 00:53:28.529237 7011 master_runner.cc:305] Clearing existing system tablet
I20260619 00:53:28.529732 7011 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:28.529836 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:28.533017 7011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 00:53:28.535056 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:28.535411 7011 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:28.535679 7011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 00:53:28.535782 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "5e34565f245edad5c88c95b358428186"
server_key_iv: "707f9329a9c74babafdbd372341a8025"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.540395 7011 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:28.548091 7011 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 00:53:28.548305 7011 log.cc:1196] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 00:53:28.548869 7011 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Deleting consensus metadata
I20260619 00:53:28.549142 7011 master_runner.cc:315] Copying system tablet from 127.0.97.62:41499
I20260619 00:53:28.551733 7011 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41499
I20260619 00:53:28.567703 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.552507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33572)
0619 00:53:28.553179 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.553209 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.553369 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.553791 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.553796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.554316 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:28.554591 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.554604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.556014 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.556020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.557600 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.557623 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.558634 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.558640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.558825 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.559544 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:28.559574 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:28.560908 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:28.564789 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.564795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.564799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.565353 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.565784 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:28.565789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:28.565793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:28.565952 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:28.566558 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:28.566563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:28.566763 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.566950 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.567263 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":401,"threads_started":1}
I20260619 00:53:28.567770 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.552541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.97.62:41499)
0619 00:53:28.552742 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.552749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.553320 (+ 571us) server_negotiation.cc:407] Connection header received
0619 00:53:28.553524 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.553530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.553631 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.553742 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:28.554744 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.555853 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.557772 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.558467 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.558598 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.561097 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:28.561104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:28.561109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:28.561159 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:28.564673 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.565491 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.565497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.565501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.565659 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:28.566070 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:28.566075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:28.566078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:28.566434 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:28.566581 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.567085 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.567343 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:53:28.569101 6570 tablet_copy_service.cc:140] P f3e506e1c3364b11af306c7f83c7186b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cd6d1e41c6a74837add03c2f4ae2b8ae ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33572)
I20260619 00:53:28.569383 6570 tablet_copy_service.cc:161] P f3e506e1c3364b11af306c7f83c7186b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cd6d1e41c6a74837add03c2f4ae2b8ae at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33572: session id = cd6d1e41c6a74837add03c2f4ae2b8ae-00000000000000000000000000000000
I20260619 00:53:28.573918 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 00:53:28.576592 7011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 00:53:28.583535 7011 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: tablet copy: Starting download of 0 data blocks...
I20260619 00:53:28.583869 7011 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: tablet copy: Starting download of 1 WAL segments...
I20260619 00:53:28.586762 7011 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 00:53:28.591584 6570 tablet_copy_service.cc:342] P f3e506e1c3364b11af306c7f83c7186b: Request end of tablet copy session cd6d1e41c6a74837add03c2f4ae2b8ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33572
I20260619 00:53:28.591854 6570 tablet_copy_service.cc:433] P f3e506e1c3364b11af306c7f83c7186b: ending tablet copy session cd6d1e41c6a74837add03c2f4ae2b8ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cd6d1e41c6a74837add03c2f4ae2b8ae
I20260619 00:53:28.604233 7011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:28.606982 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:28.608253 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:28.616080 7011 server_base.cc:1034] running on GCE node
W20260619 00:53:28.615909 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:28.616580 7011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:28.617249 7011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:28.618472 7011 hybrid_clock.cc:648] HybridClock initialized: now 1781830408618407 us; error 35 us; skew 500 ppm
I20260619 00:53:28.618714 7011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:28.620991 7011 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 00:53:28.623500 7011 webserver.cc:460] Webserver started at http://127.0.97.61:39583/ using document root <none> and password file <none>
I20260619 00:53:28.623986 7011 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:28.624081 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:28.627351 7011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 00:53:28.629235 7138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:28.629611 7011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 00:53:28.629769 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "5e34565f245edad5c88c95b358428186"
server_key_iv: "707f9329a9c74babafdbd372341a8025"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.629972 7011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:28.640784 7011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:28.645435 7011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:28.668572 7011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39429
I20260619 00:53:28.668649 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39429 every 8 connection(s)
I20260619 00:53:28.670080 7011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
I20260619 00:53:28.675787 7190 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:53:28.679819 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Bootstrap starting.
I20260619 00:53:28.694489 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:53:28.694998 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Bootstrap complete.
I20260619 00:53:28.696035 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } }
I20260619 00:53:28.696307 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd6d1e41c6a74837add03c2f4ae2b8ae, State: Initialized, Role: LEARNER
I20260619 00:53:28.696633 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } }
I20260619 00:53:28.697782 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:28.697556 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.698112 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:28.706101 7204 catalog_manager.cc:1259] Loaded cluster ID: 27d07f98f5a44951a0748fd462b1060e
I20260619 00:53:28.706241 7204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: loading cluster ID for follower catalog manager: success
I20260619 00:53:28.708634 7204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: acquiring CA information for follower catalog manager: success
I20260619 00:53:28.710556 7204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:53:28.809544 7006 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
W20260619 00:53:28.883724 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:28.884038 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:28.884127 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:28.892232 7110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:28.892349 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:28.892395 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:28.892452 7110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:28.892496 7110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:28.903061 7110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:28.904354 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:28.905766 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:28.918040 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:28.918100 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:28.918282 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:28.919099 7110 server_base.cc:1034] running on GCE node
I20260619 00:53:28.919914 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:28.921383 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:28.922642 7110 hybrid_clock.cc:648] HybridClock initialized: now 1781830408922571 us; error 65 us; skew 500 ppm
I20260619 00:53:28.923084 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:28 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:28.927426 7110 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 00:53:28.936383 7110 webserver.cc:460] Webserver started at http://127.0.97.60:45673/ using document root <none> and password file <none>
I20260619 00:53:28.937196 7110 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:28.937350 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:28.938069 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:28.941185 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/instance:
uuid: "9deb79c8cd5248d89cc594599d7686a8"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "88d8cc06090347e83f8d500bf0120cac"
server_key_iv: "b2954e24b3f42751e8857295c1ca4023"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.941957 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal/instance:
uuid: "9deb79c8cd5248d89cc594599d7686a8"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "88d8cc06090347e83f8d500bf0120cac"
server_key_iv: "b2954e24b3f42751e8857295c1ca4023"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.947093 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 00:53:28.950860 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:28.952167 7110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:53:28.952361 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "9deb79c8cd5248d89cc594599d7686a8"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "88d8cc06090347e83f8d500bf0120cac"
server_key_iv: "b2954e24b3f42751e8857295c1ca4023"
server_key_version: "encryptionkey@0"
I20260619 00:53:28.952545 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:28.965920 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.966223 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:28.973417 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.966336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.97.61:39429)
0619 00:53:28.967148 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:28.967152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:28.967170 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:53:28.967243 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:28.967246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:28.967345 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:28.967468 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:28.967928 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.969442 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.971721 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.972504 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.972610 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.972793 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:28.973064 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 00:53:28.973160 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":585,"threads_started":1}
I20260619 00:53:28.973476 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:28.966194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51552)
0619 00:53:28.966837 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:28.966858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:28.966996 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:28.967515 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:28.967535 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:28.967562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:28.967795 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.967806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.969575 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.969580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.971550 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:28.971563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:28.972677 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:28.972683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:28.972886 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:28.972945 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 00:53:28.973042 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 00:53:28.973175 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":386,"threads_started":1}
I20260619 00:53:28.974418 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e506e1c3364b11af306c7f83c7186b. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:28.974713 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:28.975317 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:28.975370 7108 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: attempting to promote NON_VOTER cd6d1e41c6a74837add03c2f4ae2b8ae to VOTER
I20260619 00:53:28.976226 7108 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } }
I20260619 00:53:28.978003 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 LEARNER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 00:53:28.978647 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 00:53:28.980532 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:28.980887 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:28.981977 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } }
I20260619 00:53:28.985282 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } }
I20260619 00:53:28.985313 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } }
I20260619 00:53:28.985579 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:28.985579 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:28.985687 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } }
I20260619 00:53:28.990235 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } }
I20260619 00:53:28.990482 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:29.004639 7110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33489
I20260619 00:53:29.004660 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33489 every 8 connection(s)
I20260619 00:53:29.006044 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
I20260619 00:53:29.010205 7274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:29.010974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7110
I20260619 00:53:29.020325 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:29.036375 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.013295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.0.97.60:33489)
0619 00:53:29.014134 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.014147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.014187 (+ 40us) server_negotiation.cc:407] Connection header received
0619 00:53:29.015636 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.015668 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.016100 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.016469 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.018383 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.019675 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.021514 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.022254 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.022393 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.027760 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.027767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.027778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.027815 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.031669 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.032468 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.032481 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.032492 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.032710 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.033226 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.033230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.033232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.033746 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.034507 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.034920 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.035368 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":390,"threads_started":1}
I20260619 00:53:29.039758 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.023799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33578)
0619 00:53:29.024599 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.024643 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.024849 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.025233 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.025240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.025852 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.026092 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.026120 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.027255 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.027260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.028659 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.028676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.029873 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.029882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.030074 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.030776 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.030793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.033531 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.036662 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.036674 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.036682 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.037311 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.037691 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.037695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.037698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.037841 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.038634 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.038642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.038842 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.039052 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.039366 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":422,"threads_started":1}
I20260619 00:53:29.040021 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.023394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.0.97.62:41499)
0619 00:53:29.023604 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.023610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.024745 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 00:53:29.024992 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.024996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.025057 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.025165 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.026264 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.027104 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.028854 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.029657 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.029803 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.033740 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.033745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.033749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.033782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.036524 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.037434 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.037438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.037441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.037573 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.037998 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.038006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.038012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.038393 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.038545 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.039415 (+ 870us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.039664 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:53:29.041175 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:29.053583 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.041875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.97.61:39429)
0619 00:53:29.042067 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.042074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.042148 (+ 74us) server_negotiation.cc:407] Connection header received
0619 00:53:29.042365 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.042371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.042453 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.042553 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.043199 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.043909 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.045195 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.045614 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.045736 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.048665 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.048671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.048674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.048709 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.051231 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.051848 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.051852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.051854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.051989 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.052383 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.052386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.052388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.052624 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.052729 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.053195 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.053363 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:53:29.053608 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.041872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51560)
0619 00:53:29.042059 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.042078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.042205 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.042577 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.042582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.042947 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.043070 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.043077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.044038 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.044043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.045063 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.045074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.045762 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.045768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.045943 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.046498 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.046516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.048465 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.051371 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.051375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.051378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.051724 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.052123 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.052127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.052130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.052274 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.052736 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.052739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.052957 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.053119 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.053361 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:53:29.054632 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830408, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 00:53:29.066315 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.055379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.0.1:59410)
0619 00:53:29.055525 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.055539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.055641 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.055975 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.055978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.056317 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.056482 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.056489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.057471 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.057475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.058286 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.058295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.059347 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.059351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.059493 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.059957 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.059972 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.061857 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.064374 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.064378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.064380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.064723 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.065060 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.065063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.065065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.065210 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.065637 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.065640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.065759 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.065862 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.066033 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 00:53:29.066453 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.055309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.0.97.60:33489)
0619 00:53:29.055477 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.055483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.055638 (+ 155us) server_negotiation.cc:407] Connection header received
0619 00:53:29.055765 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.055768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.055826 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.055939 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.056606 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.057369 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.058400 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.059204 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.059308 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.062008 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.062013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.062016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.062042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.064249 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.064850 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.064858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.064862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.064963 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.065295 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.065298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.065300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.065549 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.065652 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.065932 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.066147 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":58}
I20260619 00:53:29.071434 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Bootstrap starting.
I20260619 00:53:29.074504 7274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:29.075515 7274 log.cc:824] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:29.078403 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: No bootstrap required, opened a new log
I20260619 00:53:29.085224 7274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } }
I20260619 00:53:29.085546 7274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:29.085659 7274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9deb79c8cd5248d89cc594599d7686a8, State: Initialized, Role: FOLLOWER
I20260619 00:53:29.086191 7274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } }
I20260619 00:53:29.087006 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } } }
I20260619 00:53:29.087330 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:29.087977 7274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 00:53:29.096788 7291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:53:29.096962 7291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:53:29.101843 7110 master_runner.cc:428] Detected that this master 9deb79c8cd5248d89cc594599d7686a8 is joining an existing cluster
I20260619 00:53:29.101962 7110 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:33489
I20260619 00:53:29.132387 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.106459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.97.60:33489)
0619 00:53:29.106700 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.106710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.115314 (+ 8604us) server_negotiation.cc:407] Connection header received
0619 00:53:29.115558 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.115583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.115665 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.115783 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.117778 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.118957 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.121127 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.121979 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.122084 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.125333 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.125340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.125345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.125374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.129274 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.130152 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.130159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.130165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.130318 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.130682 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.130687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.130690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.131160 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.131295 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.131803 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.132014 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":68}
I20260619 00:53:29.133242 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.106597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.97.62:41499)
0619 00:53:29.106940 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.106946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.114144 (+ 7198us) server_negotiation.cc:407] Connection header received
0619 00:53:29.115027 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.115034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.115120 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.115192 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.117644 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.118996 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.121515 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.122285 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.122393 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.125610 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.125618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.125623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.125673 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.129526 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.130455 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.130462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.130469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.130683 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.131262 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.131268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.131272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.131712 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.131870 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.132546 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.132832 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260619 00:53:29.136626 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.105910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.0.1:59418)
0619 00:53:29.115191 (+ 9281us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.115216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.115430 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.115962 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.115967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.116847 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.117633 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.117645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.119171 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.119177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.120980 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.120993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.122377 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.122383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.122537 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.123834 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.123859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.125161 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.129418 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.129427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.129431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.130028 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.130428 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.130432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.130436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.130596 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.131352 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.131357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.131500 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.132990 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.133282 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9029,"mutex_wait_us":69,"spinlock_wait_cycles":32128,"thread_start_us":8218,"threads_started":1}
I20260619 00:53:29.139705 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.105843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33594)
0619 00:53:29.113993 (+ 8150us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.114017 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.114823 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.115425 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.115434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.116802 (+ 1368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.117467 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.117484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.119249 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.119255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.121309 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.121330 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.122736 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.122746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.123023 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.123938 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.123970 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.125379 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.129692 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.129750 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.129758 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.130308 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.130901 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.130908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.130914 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.131124 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.131952 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.131958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.132212 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.133687 (+ 1475us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.139192 (+ 5505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7999,"mutex_wait_us":144,"thread_start_us":7838,"threads_started":1}
I20260619 00:53:29.144625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6611
I20260619 00:53:29.148710 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.106397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.0.97.61:39429)
0619 00:53:29.106801 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.106807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.117926 (+ 11119us) server_negotiation.cc:407] Connection header received
0619 00:53:29.118152 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.118158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.118240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.118307 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.119888 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.120964 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.123753 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.124442 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.124574 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.138950 (+ 14376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.138957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.138962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.139005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.142470 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.145814 (+ 3344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.145820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.145823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.145994 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.146915 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.146921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.146924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.147314 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.147457 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.148083 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.148337 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260619 00:53:29.149887 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.105915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:51564)
0619 00:53:29.117823 (+ 11908us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.117843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.118014 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.118491 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.118495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.119221 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.119693 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.119704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.121136 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.121140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.123150 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.123174 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.133434 (+ 10260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.133441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.133652 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.137520 (+ 3868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.137542 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.138677 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.145139 (+ 6462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.145146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.145152 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.145653 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.146148 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.146152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.146156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.146774 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.147541 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.147546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.147764 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.149089 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.149405 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11761,"mutex_wait_us":84,"spinlock_wait_cycles":23808,"thread_start_us":11047,"threads_started":1}
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830406, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 00:53:29.162904 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.147962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.97.60:33489)
0619 00:53:29.148120 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.148126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.148141 (+ 15us) server_negotiation.cc:407] Connection header received
0619 00:53:29.148236 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.148243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.148347 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.148438 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.149693 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.150732 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.152203 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.152653 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.152800 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.155963 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.155973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.155980 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.156029 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.159923 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.160880 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.160886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.160890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.161030 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.161408 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.161411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.161413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.161681 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.161838 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.162217 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.162443 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:53:29.162897 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.147303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.0.1:59422)
0619 00:53:29.147654 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.147674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.147771 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.148674 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.148679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.149206 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.149388 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.149399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.151033 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.151040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.152057 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.152070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.152841 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.152849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.153021 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.153639 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.153656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.155724 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.160099 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.160105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.160109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.160689 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.161162 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.161166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.161168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.161290 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.161823 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.161828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.161979 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.162125 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.162411 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260619 00:53:29.164777 6550 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:33489: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 33489
}
attrs {
promote: true
}
}
dest_uuid: "f3e506e1c3364b11af306c7f83c7186b"
cas_config_opid_index: 6
I20260619 00:53:29.165639 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } }
I20260619 00:53:29.168036 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 00:53:29.168973 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 00:53:29.169433 7249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:29.171676 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.173470 7249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 00:53:29.175799 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:29.176002 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.178776 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.146025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.97.1:36365)
0619 00:53:29.156436 (+ 10411us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.156446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.156478 (+ 32us) server_negotiation.cc:407] Connection header received
0619 00:53:29.156551 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.156561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.156958 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.157144 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.158868 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.161000 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.163096 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.164061 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.164207 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.170228 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.170247 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.170276 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.170328 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.174369 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.175305 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.175316 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.175320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.175479 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.175970 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.175975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.175979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.176436 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.177850 (+ 1414us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.178023 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.178355 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10210,"thread_start_us":10041,"threads_started":1}
I20260619 00:53:29.180215 7109 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:33489
I20260619 00:53:29.180959 7110 master.cc:537] Master@127.0.97.60:33489 shutting down...
I20260619 00:53:29.181468 6734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:29.182179 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.182502 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:29.182972 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.183279 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.183598 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.183876 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.198583 7249 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } }
W20260619 00:53:29.198824 7249 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } }
I20260619 00:53:29.244516 7249 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:29.250164 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e506e1c3364b11af306c7f83c7186b. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:29.250142 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:29.250432 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 00:53:29.250496 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 00:53:29.249677 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: true } } }
I20260619 00:53:29.250893 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 00:53:29.252264 7109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.252537 7109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
W20260619 00:53:29.253104 6535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 00:53:29.254208 7110 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 00:53:29.254703 7110 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 00:53:29.254876 7110 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: stopping tablet replica
W20260619 00:53:29.257047 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b -> Peer 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60:33489): Couldn't send request to peer 9deb79c8cd5248d89cc594599d7686a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 00:53:29.263084 7110 master.cc:552] Master@127.0.97.60:33489 shutdown complete.
I20260619 00:53:29.263180 7110 master_runner.cc:305] Clearing existing system tablet
I20260619 00:53:29.263571 7110 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:29.263680 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:29.266644 7110 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 00:53:29.273212 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:29.273712 7110 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:29.274155 7110 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.007s
I20260619 00:53:29.274307 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "9deb79c8cd5248d89cc594599d7686a8"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "88d8cc06090347e83f8d500bf0120cac"
server_key_iv: "b2954e24b3f42751e8857295c1ca4023"
server_key_version: "encryptionkey@0"
I20260619 00:53:29.279323 7110 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:29.286971 7110 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 00:53:29.287282 7110 log.cc:1196] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 00:53:29.287863 7110 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Deleting consensus metadata
I20260619 00:53:29.288188 7110 master_runner.cc:315] Copying system tablet from 127.0.97.62:41499
I20260619 00:53:29.291054 7110 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:41499
I20260619 00:53:29.308070 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.292319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.97.62:41499)
0619 00:53:29.292535 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.292542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.292984 (+ 442us) server_negotiation.cc:407] Connection header received
0619 00:53:29.293255 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.293262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.293349 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.293413 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.294548 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.295695 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.297651 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.298315 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.298654 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.301264 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.301271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.301276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.301315 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.305088 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.305880 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.305887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.305891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.306037 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.306481 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.306486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.306490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.306822 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.306952 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.307499 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.307719 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 00:53:29.309341 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.291930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:33610)
0619 00:53:29.292850 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.292875 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.293111 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.293602 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.293607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.294137 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.294382 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.294394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.295876 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.295881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.297486 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.297504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.298863 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.298869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.299035 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.299766 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.299787 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.301008 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.305254 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.305260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.305264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.305740 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.306189 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.306193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.306197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.306365 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.307031 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.307036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.307224 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.308588 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.308940 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":639,"threads_started":1}
I20260619 00:53:29.310194 6570 tablet_copy_service.cc:140] P f3e506e1c3364b11af306c7f83c7186b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9deb79c8cd5248d89cc594599d7686a8 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33610)
I20260619 00:53:29.310376 6570 tablet_copy_service.cc:161] P f3e506e1c3364b11af306c7f83c7186b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9deb79c8cd5248d89cc594599d7686a8 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33610: session id = 9deb79c8cd5248d89cc594599d7686a8-00000000000000000000000000000000
I20260619 00:53:29.314440 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 00:53:29.315939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6611
I20260619 00:53:29.316236 7110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 00:53:29.324262 7110 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: tablet copy: Starting download of 0 data blocks...
I20260619 00:53:29.324616 7110 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: tablet copy: Starting download of 1 WAL segments...
I20260619 00:53:29.327265 7110 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 00:53:29.332655 6570 tablet_copy_service.cc:342] P f3e506e1c3364b11af306c7f83c7186b: Request end of tablet copy session 9deb79c8cd5248d89cc594599d7686a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33610
I20260619 00:53:29.332871 6570 tablet_copy_service.cc:433] P f3e506e1c3364b11af306c7f83c7186b: ending tablet copy session 9deb79c8cd5248d89cc594599d7686a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9deb79c8cd5248d89cc594599d7686a8
I20260619 00:53:29.339116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36365
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41499
--webserver_port=36623
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:53:29.345297 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:29.348537 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:29.349905 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:29.357322 7110 server_base.cc:1034] running on GCE node
W20260619 00:53:29.358146 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:29.358734 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:29.359462 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:29.360671 7110 hybrid_clock.cc:648] HybridClock initialized: now 1781830409360632 us; error 36 us; skew 500 ppm
I20260619 00:53:29.360957 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:29.363449 7110 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 00:53:29.368356 7110 webserver.cc:460] Webserver started at http://127.0.97.60:36021/ using document root <none> and password file <none>
I20260619 00:53:29.368707 7110 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:29.368841 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:29.371868 7110 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 00:53:29.373901 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:29.374325 7110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 00:53:29.374445 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "9deb79c8cd5248d89cc594599d7686a8"
format_stamp: "Formatted at 2026-06-19 00:53:28 on dist-test-slave-r7bh"
server_key: "88d8cc06090347e83f8d500bf0120cac"
server_key_iv: "b2954e24b3f42751e8857295c1ca4023"
server_key_version: "encryptionkey@0"
I20260619 00:53:29.374639 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:29.383488 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:29.387408 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:29.431298 7110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33489
I20260619 00:53:29.431370 7379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33489 every 8 connection(s)
I20260619 00:53:29.432531 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
I20260619 00:53:29.437939 7380 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:53:29.441859 7380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Bootstrap starting.
I20260619 00:53:29.456898 7380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:53:29.457417 7380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: Bootstrap complete.
I20260619 00:53:29.458498 7380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } }
I20260619 00:53:29.458843 7380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9deb79c8cd5248d89cc594599d7686a8, State: Initialized, Role: LEARNER
I20260619 00:53:29.459127 7380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } }
I20260619 00:53:29.459975 7383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.460407 7383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:29.460487 7380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:29.469466 7394 catalog_manager.cc:1259] Loaded cluster ID: 27d07f98f5a44951a0748fd462b1060e
I20260619 00:53:29.469676 7394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: loading cluster ID for follower catalog manager: success
I20260619 00:53:29.473474 7394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: acquiring CA information for follower catalog manager: success
I20260619 00:53:29.475983 7394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 00:53:29.724067 7317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:29.724488 7317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:29.724665 7317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:29.732643 7317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:29.732798 7317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:29.733009 7317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:29.743057 7317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36623
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:29.744470 7317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:29.745927 7317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:29.760128 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:29.760206 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:29.760102 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:29.761036 7317 server_base.cc:1034] running on GCE node
I20260619 00:53:29.761720 7317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:29.762717 7317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:29.763947 7317 hybrid_clock.cc:648] HybridClock initialized: now 1781830409763892 us; error 46 us; skew 500 ppm
I20260619 00:53:29.764326 7317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:29.768505 7317 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 00:53:29.770195 7317 webserver.cc:460] Webserver started at http://127.0.97.1:36623/ using document root <none> and password file <none>
I20260619 00:53:29.770884 7317 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:29.771004 7317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:29.776707 7317 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:53:29.779377 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:29.781018 7317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:53:29.781206 7317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51"
format_stamp: "Formatted at 2026-06-19 00:53:26 on dist-test-slave-r7bh"
server_key: "7df5f21f0a558889c233c3df889e4ca6"
server_key_iv: "f4d44ed74b7b673b53f3b78e048a2efd"
server_key_version: "encryptionkey@0"
I20260619 00:53:29.782516 7317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:29.793105 7414 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.799981 7414 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.807905 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.793451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.97.60:33489)
0619 00:53:29.801979 (+ 8528us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.801985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.802004 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:53:29.802075 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.802088 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.802182 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.802335 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:29.802863 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.803826 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.806041 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.806843 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.807013 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.807289 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.807433 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.807567 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8378,"thread_start_us":8266,"threads_started":1}
I20260619 00:53:29.807909 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.793413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.0.1:59430)
0619 00:53:29.801232 (+ 7819us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.801261 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.801473 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.802355 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.802360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.802390 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:29.802631 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.802647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.803957 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.803963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.805870 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.805886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.807037 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.807043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.807222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.807266 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.807409 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.807555 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7616,"thread_start_us":7524,"threads_started":1}
I20260619 00:53:29.808779 7355 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260619 00:53:29.809247 7383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e506e1c3364b11af306c7f83c7186b. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: true } } }
I20260619 00:53:29.809561 7383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: LEARNER
I20260619 00:53:29.809988 7415 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: attempting to promote NON_VOTER 9deb79c8cd5248d89cc594599d7686a8 to VOTER
I20260619 00:53:29.810645 7415 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } }
I20260619 00:53:29.811033 7317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:29.812461 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 00:53:29.812650 7355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 LEARNER]: Refusing update from remote peer f3e506e1c3364b11af306c7f83c7186b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 00:53:29.813078 7415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 00:53:29.813333 7416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260619 00:53:29.816612 7317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:29.816470 7416 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.816915 7317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:29.817049 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.818176 7317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:29.819907 7317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:29.820024 7317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:29.820147 7317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:29.820231 7317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:29.821712 7416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.821971 7416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.822392 7416 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.822633 7416 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.821878 7355 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.823652 7425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.824014 7425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:29.828281 7383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.828624 7383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:29.828919 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e506e1c3364b11af306c7f83c7186b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:29.829173 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:29.862703 7317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36365
I20260619 00:53:29.862757 7531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36365 every 8 connection(s)
I20260619 00:53:29.864485 7317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:29.871822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7317
I20260619 00:53:29.872218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6745
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:29.897917 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.867467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41251 (local address 127.0.97.60:33489)
0619 00:53:29.867732 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.867742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.879226 (+ 11484us) server_negotiation.cc:407] Connection header received
0619 00:53:29.879387 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.879392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.879474 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.879566 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.880479 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.881646 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.884221 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.884866 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.884955 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.890294 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.890300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.890305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.890340 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.893972 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895133 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.895142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.895145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.895283 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895786 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.895793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.895798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.896147 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.896272 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.897281 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.897511 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 00:53:29.898092 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.867457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36431 (local address 127.0.97.62:41499)
0619 00:53:29.868027 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.868033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.870124 (+ 2091us) server_negotiation.cc:407] Connection header received
0619 00:53:29.873771 (+ 3647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.873780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.873895 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.874016 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.876879 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.878450 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.881418 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.882337 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.882488 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.889642 (+ 7154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.889651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.889655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.889696 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.894089 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895121 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.895126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.895130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.895268 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895764 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.895768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.895771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.896125 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.896265 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.897511 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.897713 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
I20260619 00:53:29.898438 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.868555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50835 (local address 127.0.97.61:39429)
0619 00:53:29.869673 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.869682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.880514 (+ 10832us) server_negotiation.cc:407] Connection header received
0619 00:53:29.880583 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.880590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.880666 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.880767 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.882492 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.883636 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.886069 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.886679 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.886774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.891203 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.891212 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.891219 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.891275 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.895173 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895929 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.895935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.895938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.896062 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.896508 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.896516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.896519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.896852 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.896998 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.897921 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.898126 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":757,"threads_started":1}
Jun 19 00:53:29 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 00:53:29.900286 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.868745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.97.1:41251)
0619 00:53:29.879115 (+ 10370us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.879133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.879273 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.879584 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.879588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.880099 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.880342 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.880349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.881840 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.881845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.884054 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.884065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.885234 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.885241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.885466 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.886144 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.886165 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.890067 (+ 3902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.894147 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.894157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.894176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.894989 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.895445 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.895454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.895638 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.896378 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.896383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.896847 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.899082 (+ 2235us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.899363 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10272,"mutex_wait_us":33,"spinlock_wait_cycles":10496,"thread_start_us":10606,"threads_started":1}
I20260619 00:53:29.900324 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.868166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:36431)
0619 00:53:29.869940 (+ 1774us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.869986 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.873271 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.874600 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.874611 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.875557 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.876572 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.876601 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.878619 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.878627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.881235 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.881249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.882907 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.882916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.883757 (+ 841us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.884904 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.884991 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.889378 (+ 4387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.894258 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.894262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.894266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.894976 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.895384 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.895397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.895621 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.896335 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.896343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.896846 (+ 503us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.897183 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.898728 (+ 1545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":1174,"spinlock_wait_cycles":32640,"thread_start_us":1279,"threads_started":1}
I20260619 00:53:29.900405 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.868167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.97.1:50835)
0619 00:53:29.878016 (+ 9849us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:29.878040 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:29.878174 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:29.881456 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:29.881461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:29.881979 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:29.882318 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.882329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.883837 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.883842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.885912 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:29.885928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.887118 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.887124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:29.887335 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.887970 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:29.887991 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:29.890780 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:29.895326 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.895333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.895336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.895804 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.896191 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:29.896195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:29.896199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:29.896391 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:29.897053 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:29.897057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:29.897183 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 00:53:29.897326 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 00:53:29.897701 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9679,"mutex_wait_us":512,"spinlock_wait_cycles":17792,"thread_start_us":9934,"threads_started":1}
I20260619 00:53:29.902069 7533 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:29.902544 7532 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33489
I20260619 00:53:29.902547 7533 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:29.902618 7534 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39429
I20260619 00:53:29.902710 7532 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:29.902779 7534 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:29.903746 7532 heartbeater.cc:511] Master 127.0.97.60:33489 requested a full tablet report, sending...
I20260619 00:53:29.903784 7534 heartbeater.cc:511] Master 127.0.97.61:39429 requested a full tablet report, sending...
I20260619 00:53:29.903811 7533 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
I20260619 00:53:29.905491 6550 ts_manager.cc:194] Re-registered known tserver with Master: bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365)
I20260619 00:53:29.905990 7155 ts_manager.cc:194] Registered new tserver with Master: bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365)
I20260619 00:53:29.906345 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36431
I20260619 00:53:29.907655 7345 ts_manager.cc:194] Registered new tserver with Master: bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365)
I20260619 00:53:29.909809 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:29.873842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.97.2:40347)
0619 00:53:29.881368 (+ 7526us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:29.881379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:29.886010 (+ 4631us) server_negotiation.cc:407] Connection header received
0619 00:53:29.886299 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:29.886310 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:29.886752 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:29.886894 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:29.888856 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.890552 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.892343 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:29.893283 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:29.893416 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:29.899735 (+ 6319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:29.899756 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:29.899795 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:29.899840 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:29.906832 (+ 6992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.907570 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.907584 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.907586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.907714 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:29.908153 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:29.908158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:29.908161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:29.908522 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:29.908813 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:29.909190 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 00:53:29.909458 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7370,"thread_start_us":7222,"threads_started":1}
I20260619 00:53:29.910946 6868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:29.996623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6745
I20260619 00:53:30.015275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40347
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41499
--webserver_port=41547
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:30.363721 7544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:30.364110 7544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:30.364284 7544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:30.372054 7544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:30.372188 7544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:30.372310 7544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:30.382982 7544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41547
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:30.384325 7544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:30.385701 7544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:30.399797 7552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:30.399822 7550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:30.399871 7549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:30.401053 7544 server_base.cc:1034] running on GCE node
I20260619 00:53:30.401780 7544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:30.402925 7544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:30.404217 7544 hybrid_clock.cc:648] HybridClock initialized: now 1781830410404098 us; error 40 us; skew 500 ppm
I20260619 00:53:30.404657 7544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:30 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:30.408293 7544 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 00:53:30.409924 7544 webserver.cc:460] Webserver started at http://127.0.97.2:41547/ using document root <none> and password file <none>
I20260619 00:53:30.410699 7544 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:30.410821 7544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:30.416718 7544 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:53:30.419548 7559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:30.421160 7544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:53:30.421322 7544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "6807ccc12ab74177a2774fc89f23df1b"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "381fa2483f144a6cfa72aaa42bbfbd46"
server_key_iv: "253a99133d68d5abf7aa6e71b52368d1"
server_key_version: "encryptionkey@0"
I20260619 00:53:30.422348 7544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:30.447924 7544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:30.452472 7544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:30.452714 7544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:30.453725 7544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:30.455250 7544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:30.455343 7544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:30.455416 7544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:30.455470 7544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:30.490082 7544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40347
I20260619 00:53:30.490160 7671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40347 every 8 connection(s)
I20260619 00:53:30.491629 7544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:30.497380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7544
I20260619 00:53:30.497752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6878
Jun 19 00:53:30 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 00:53:30 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 00:53:30.553544 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.495005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43379 (local address 127.0.97.62:41499)
0619 00:53:30.522523 (+ 27518us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:30.522533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:30.522570 (+ 37us) server_negotiation.cc:407] Connection header received
0619 00:53:30.522653 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:30.522662 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:30.522776 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:30.522938 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:30.528584 (+ 5646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.529860 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.533228 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.534084 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.534221 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.545415 (+ 11194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:30.545423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:30.545428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:30.545479 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:30.549687 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.550969 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.550976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.550980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.551125 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.551625 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.551630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.551634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.551898 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:30.552051 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:30.552933 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 00:53:30.553170 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27337,"thread_start_us":29827,"threads_started":1}
Jun 19 00:53:30 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 00:53:30.554126 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.495119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45517 (local address 127.0.97.60:33489)
0619 00:53:30.524363 (+ 29244us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:30.524373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:30.524400 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:53:30.524477 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:30.524484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:30.524593 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:30.524734 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:30.528572 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.530183 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.533230 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.534159 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.534289 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.545781 (+ 11492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:30.545787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:30.545792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:30.545836 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:30.549719 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.551004 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.551010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.551013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.551178 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.551671 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.551677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.551682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.552049 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:30.552202 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:30.553444 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0619 00:53:30.553702 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29071,"thread_start_us":28935,"threads_started":1}
I20260619 00:53:30.555449 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.495789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.97.2:45517)
0619 00:53:30.523661 (+ 27872us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:30.523689 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:30.523839 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:30.525230 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:30.525249 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:30.526819 (+ 1570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:30.528235 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.528274 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.530437 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.530445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.532997 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.533019 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.534718 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.534728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.535359 (+ 631us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.536518 (+ 1159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:30.536582 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:30.545525 (+ 8943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:30.549925 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.549936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.549956 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.550846 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.551339 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.551345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.551350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.551536 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.552294 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:30.552298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:30.552493 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 00:53:30.554367 (+ 1874us) client_negotiation.cc:240] Negotiation successful
0619 00:53:30.554678 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27635,"mutex_wait_us":484,"spinlock_wait_cycles":101888,"thread_start_us":35862,"threads_started":1}
I20260619 00:53:30.555464 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.495789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:43379)
0619 00:53:30.513238 (+ 17449us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:30.513310 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:30.521313 (+ 8003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:30.525228 (+ 3915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:30.525249 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:30.526798 (+ 1549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:30.528235 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.528274 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.530131 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.530140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.532994 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.533016 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.534673 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.534689 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.535351 (+ 662us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.536527 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:30.536577 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:30.545089 (+ 8512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:30.550179 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.550186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.550192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.550842 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.551241 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.551247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.551251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.551519 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.552032 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:30.552040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:30.552495 (+ 455us) client_negotiation.cc:769] Sending connection context
0619 00:53:30.552852 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 00:53:30.553417 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16953,"mutex_wait_us":257,"spinlock_wait_cycles":31104,"thread_start_us":27320,"threads_started":1}
I20260619 00:53:30.557397 7673 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:30.557767 7673 heartbeater.cc:464] Registering TS with master...
Jun 19 00:53:30 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 00:53:30.558840 7673 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
I20260619 00:53:30.559891 7672 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33489
I20260619 00:53:30.560035 7672 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:30.560566 7672 heartbeater.cc:511] Master 127.0.97.60:33489 requested a full tablet report, sending...
I20260619 00:53:30.561105 6550 ts_manager.cc:194] Re-registered known tserver with Master: 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:30.561882 7345 ts_manager.cc:194] Registered new tserver with Master: 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:30.562299 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43379
I20260619 00:53:30.566697 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.499231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.97.3:43991)
0619 00:53:30.541190 (+ 41959us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:30.541201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:30.541564 (+ 363us) server_negotiation.cc:407] Connection header received
0619 00:53:30.541810 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:30.541822 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:30.542156 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:30.542340 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:30.543914 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.545807 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.547502 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.548289 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.548422 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.557110 (+ 8688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:30.557130 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:30.557147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:30.557205 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:30.561984 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.563021 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.563031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.563035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.563222 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.564075 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.564080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.564084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.564507 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:30.564792 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:30.566025 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0619 00:53:30.566294 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41720,"thread_start_us":42018,"threads_started":1}
I20260619 00:53:30.568961 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.511243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.97.2:45575)
0619 00:53:30.545110 (+ 33867us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:30.545133 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:30.546342 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:30.547719 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:30.547726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:30.548343 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:30.548674 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.548689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.550159 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.550166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.552680 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:30.552693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.554891 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.554898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:30.555082 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.555718 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:30.555744 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:30.558568 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:30.566214 (+ 7646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.566220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.566224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.566590 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.567057 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:30.567063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:30.567067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:30.567247 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:30.567907 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:30.567911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:30.568053 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 00:53:30.568186 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 00:53:30.568502 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33690,"thread_start_us":32629,"threads_started":1}
I20260619 00:53:30.569825 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:30.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45575 (local address 127.0.97.61:39429)
0619 00:53:30.539914 (+ 27888us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:30.539921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:30.546227 (+ 6306us) server_negotiation.cc:407] Connection header received
0619 00:53:30.546501 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:30.546508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:30.546602 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:30.546720 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:30.548900 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.550009 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.553820 (+ 3811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:30.554468 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:30.554591 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:30.558867 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:30.558876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:30.558881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:30.558927 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:30.566057 (+ 7130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.566731 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.566738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.566741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.566911 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:30.567382 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:30.567387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:30.567389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:30.567771 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:30.567885 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:30.568262 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 00:53:30.568509 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27708,"thread_start_us":27886,"threads_started":1}
I20260619 00:53:30.570863 7674 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39429
I20260619 00:53:30.571020 7674 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:30.571519 7674 heartbeater.cc:511] Master 127.0.97.61:39429 requested a full tablet report, sending...
I20260619 00:53:30.571884 7001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:30.572768 7155 ts_manager.cc:194] Registered new tserver with Master: 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:30.656423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6878
I20260619 00:53:30.673749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43991
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41499
--webserver_port=39783
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:53:30.910014 7533 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
W20260619 00:53:31.008499 7685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:31.008905 7685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:31.009097 7685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:31.016649 7685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:31.016819 7685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:31.017021 7685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:31.027184 7685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39783
--tserver_master_addrs=127.0.97.62:41499,127.0.97.61:39429,127.0.97.60:33489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:31.028628 7685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:31.030103 7685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:31.043423 7690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:31.043609 7691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:31.043929 7693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:31.044448 7685 server_base.cc:1034] running on GCE node
I20260619 00:53:31.045142 7685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:31.046063 7685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:31.047303 7685 hybrid_clock.cc:648] HybridClock initialized: now 1781830411047253 us; error 41 us; skew 500 ppm
I20260619 00:53:31.047628 7685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:31 dist-test-slave-r7bh krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:31.051098 7685 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 00:53:31.052932 7685 webserver.cc:460] Webserver started at http://127.0.97.3:39783/ using document root <none> and password file <none>
I20260619 00:53:31.053725 7685 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:31.053915 7685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:31.060730 7685 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 00:53:31.063912 7700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:31.065483 7685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:31.065687 7685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
format_stamp: "Formatted at 2026-06-19 00:53:27 on dist-test-slave-r7bh"
server_key: "da1c9abb05d7fe5f8d4337ccfdfa53c1"
server_key_iv: "c4b521f18c8fac598553caa4e1d804a9"
server_key_version: "encryptionkey@0"
I20260619 00:53:31.067101 7685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:31.083915 7685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:31.088433 7685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:31.088725 7685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:31.089792 7685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:31.091198 7685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:31.091346 7685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:31.091470 7685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:31.091558 7685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:31.125607 7685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43991
I20260619 00:53:31.125680 7812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43991 every 8 connection(s)
I20260619 00:53:31.127326 7685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:31.135902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7685
Jun 19 00:53:31 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 00:53:31 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 00:53:31 dist-test-slave-r7bh krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:31.176640 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.132177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35357 (local address 127.0.97.61:39429)
0619 00:53:31.149530 (+ 17353us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.149539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.152086 (+ 2547us) server_negotiation.cc:407] Connection header received
0619 00:53:31.152151 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.152155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.152217 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.152333 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.154964 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156073 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.158922 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.159550 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.160936 (+ 1386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.166177 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.166186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.166193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.166262 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.170495 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.171664 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.171671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.171675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.171823 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.172512 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.172521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.172527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.172950 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.173132 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.175496 (+ 2364us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.175763 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17239,"thread_start_us":18035,"threads_started":1}
I20260619 00:53:31.178952 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.131038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:52859)
0619 00:53:31.149605 (+ 18567us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.149677 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.151709 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.153158 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.153162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.154019 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:31.154759 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.154784 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156261 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.156267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.158766 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.158780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.164219 (+ 5439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.164235 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.164427 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.165080 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:31.165101 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:31.169154 (+ 4053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:31.175226 (+ 6072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.175236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.175255 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.175912 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.176556 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.176564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.176570 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.176750 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.177506 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:31.177514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:31.177763 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.177922 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.178332 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18206,"mutex_wait_us":176,"spinlock_wait_cycles":15488,"thread_start_us":21766,"threads_started":1}
I20260619 00:53:31.179383 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.131877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52859 (local address 127.0.97.62:41499)
0619 00:53:31.152791 (+ 20914us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.152797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.152840 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:53:31.152898 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.152904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.152977 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.153395 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.154963 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156073 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.161491 (+ 5418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.162294 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.162448 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.170062 (+ 7614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.170072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.170079 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.170127 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.175013 (+ 4886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.176200 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.176209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.176216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.176405 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.176911 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.176915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.176918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.177321 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.178494 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.178699 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.178995 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20788,"thread_start_us":20702,"threads_started":1}
I20260619 00:53:31.178373 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.131051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.97.3:35357)
0619 00:53:31.150010 (+ 18959us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.150026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.151709 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.152642 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.152655 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.153672 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:31.154759 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.154784 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156279 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.156286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.158766 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.158780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.159798 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.159805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.160085 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.160902 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:31.160936 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:31.165902 (+ 4966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:31.170731 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.170744 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.170772 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.171504 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.171977 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.171985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.171991 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.172344 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.173207 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:31.173218 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:31.173752 (+ 534us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.174087 (+ 335us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.174760 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18830,"mutex_wait_us":142,"spinlock_wait_cycles":42368,"thread_start_us":19086,"threads_started":1}
I20260619 00:53:31.181059 7814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41499
I20260619 00:53:31.181087 7815 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39429
I20260619 00:53:31.181478 7815 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:31.181483 7814 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:31.182473 7815 heartbeater.cc:511] Master 127.0.97.61:39429 requested a full tablet report, sending...
I20260619 00:53:31.182868 7814 heartbeater.cc:511] Master 127.0.97.62:41499 requested a full tablet report, sending...
I20260619 00:53:31.184651 7155 ts_manager.cc:194] Registered new tserver with Master: 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:31.186373 6550 ts_manager.cc:194] Re-registered known tserver with Master: 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:31.187566 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52859
I20260619 00:53:31.197000 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.131039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.97.3:46819)
0619 00:53:31.153129 (+ 22090us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.153149 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.153402 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.154061 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.154069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.154893 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:31.155248 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.155263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156972 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.156980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.159152 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.159167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.160180 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.160190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.160503 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.161266 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:31.161291 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:31.165639 (+ 4348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:31.191501 (+ 25862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.191506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.191511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.192513 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.193107 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:31.193113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:31.193116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:31.193312 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:31.195782 (+ 2470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:31.195788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:31.195938 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.196120 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.196472 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21927,"mutex_wait_us":2098,"thread_start_us":15924,"threads_started":1}
I20260619 00:53:31.198331 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.132212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46819 (local address 127.0.97.60:33489)
0619 00:53:31.146147 (+ 13935us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.146152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.153275 (+ 7123us) server_negotiation.cc:407] Connection header received
0619 00:53:31.153569 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.153577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.153691 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.153790 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.155470 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.156774 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.159313 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.159952 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.160074 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.171415 (+ 11341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.171422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.171428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.171470 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.186387 (+ 14917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.192736 (+ 6349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.192743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.192746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.192950 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.193489 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.193496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.193501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.193899 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.197429 (+ 3530us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.197565 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.197878 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13826,"thread_start_us":13923,"threads_started":1}
I20260619 00:53:31.199306 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33489
I20260619 00:53:31.199481 7813 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:31.199954 7813 heartbeater.cc:511] Master 127.0.97.60:33489 requested a full tablet report, sending...
I20260619 00:53:31.201259 7345 ts_manager.cc:194] Registered new tserver with Master: 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:31.204381 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.160439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.97.62:41499)
0619 00:53:31.169862 (+ 9423us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.169871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.169896 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:53:31.179678 (+ 9782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.179684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.179774 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.179879 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.183254 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.184321 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.190696 (+ 6375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.191112 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.191252 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.196119 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.196127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.196133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.196170 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.200549 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.201791 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.201796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.201800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.201958 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.202539 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.202544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.202548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.202900 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.203136 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.203561 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.203804 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8747,"thread_start_us":8594,"threads_started":1}
I20260619 00:53:31.211629 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.161640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.0.97.61:39429)
0619 00:53:31.169408 (+ 7768us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.169417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.185043 (+ 15626us) server_negotiation.cc:407] Connection header received
0619 00:53:31.186771 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.186777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.186855 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.186933 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.196458 (+ 9525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.197702 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.199823 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.200582 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.200684 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.204380 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.204385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.204389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.204421 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.208194 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.209273 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.209279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.209283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.209446 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.209914 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.209919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.209923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.210266 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.210489 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.210951 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.211192 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6502,"thread_start_us":6379,"threads_started":1}
I20260619 00:53:31.214447 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.164510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.97.60:33489)
0619 00:53:31.196527 (+ 32017us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.196535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.196552 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:31.196617 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.196621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.196710 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.196839 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.199168 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.200291 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.203397 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.204002 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.204114 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.207608 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.207616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.207621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.207663 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.210995 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.211846 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.211851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.211854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.212009 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.212937 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.212945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.212948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.213288 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.213538 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.213835 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.214061 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31897,"thread_start_us":31764,"threads_started":1}
I20260619 00:53:31.208860 6550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:31.217783 6550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:31.272158 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.237241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.97.3:43991)
0619 00:53:31.262934 (+ 25693us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.262944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.262979 (+ 35us) server_negotiation.cc:407] Connection header received
0619 00:53:31.263091 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.263099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.263713 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.263970 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:31.264535 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.266755 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.269419 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.270962 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.271121 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.271368 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.271645 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.271822 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25497,"thread_start_us":27035,"threads_started":1}
I20260619 00:53:31.272188 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.237597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43991 (local address 127.0.0.1:42682)
0619 00:53:31.262132 (+ 24535us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.262148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.262247 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.263941 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.263946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.263971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:31.264362 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.264375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.267022 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.267026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.269247 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.269260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.271157 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.271164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.271430 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.271474 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.271575 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.271723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24434,"mutex_wait_us":17,"thread_start_us":24690,"threads_started":1}
I20260619 00:53:31.274016 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.238441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.0.97.1:36365)
0619 00:53:31.261442 (+ 23001us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.261455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.265035 (+ 3580us) server_negotiation.cc:407] Connection header received
0619 00:53:31.265183 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.265192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.265531 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.266230 (+ 699us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:31.267426 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.268773 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.271177 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.272848 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.272942 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.273095 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.273536 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.273736 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22737,"thread_start_us":23028,"threads_started":1}
I20260619 00:53:31.274482 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.237932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36365 (local address 127.0.0.1:51968)
0619 00:53:31.264870 (+ 26938us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.264891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.265024 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.266529 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.266537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.266576 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:31.267098 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.267113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.268977 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.268980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.270969 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.271008 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.273235 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.273239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.273358 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.273379 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.273490 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.273660 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26770,"thread_start_us":24375,"threads_started":1}
I20260619 00:53:31.276537 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.236566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.0.1:44946)
0619 00:53:31.261968 (+ 25402us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.261997 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.262174 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.268132 (+ 5958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.268138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.268188 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:31.268563 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.268573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.270643 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.270648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.273966 (+ 3318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.273979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.275769 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.275775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.275946 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.275978 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.276046 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.276169 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25224,"mutex_wait_us":1,"thread_start_us":27989,"threads_started":1}
I20260619 00:53:31.276681 7748 tablet_service.cc:1467] Processing CreateTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 (DEFAULT_TABLE table=test-table [id=31d312d25ab74be591d199bcc39b2f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:31.277520 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.236797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.97.2:40347)
0619 00:53:31.267518 (+ 30721us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.267527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.267557 (+ 30us) server_negotiation.cc:407] Connection header received
0619 00:53:31.267650 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.267659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.267908 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.268764 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:31.268965 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.270430 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.274181 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.275530 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.276804 (+ 1274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.277036 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.277105 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.277264 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30528,"thread_start_us":30425,"threads_started":1}
I20260619 00:53:31.278208 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ffc1c2c7a941f58f45d4b2899e30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:31.278199 7467 tablet_service.cc:1467] Processing CreateTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 (DEFAULT_TABLE table=test-table [id=31d312d25ab74be591d199bcc39b2f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:31.279402 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ffc1c2c7a941f58f45d4b2899e30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:31.284551 7607 tablet_service.cc:1467] Processing CreateTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 (DEFAULT_TABLE table=test-table [id=31d312d25ab74be591d199bcc39b2f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:31.286011 7607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ffc1c2c7a941f58f45d4b2899e30b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:31.291800 7849 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Bootstrap starting.
I20260619 00:53:31.295145 7849 tablet_bootstrap.cc:654] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:31.295727 7850 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Bootstrap starting.
I20260619 00:53:31.296890 7849 log.cc:824] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:31.299893 7850 tablet_bootstrap.cc:654] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:31.300834 7849 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: No bootstrap required, opened a new log
I20260619 00:53:31.301119 7849 ts_tablet_manager.cc:1397] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 00:53:31.301607 7850 log.cc:824] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:31.305239 7850 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: No bootstrap required, opened a new log
I20260619 00:53:31.305522 7850 ts_tablet_manager.cc:1397] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 00:53:31.306861 7852 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Bootstrap starting.
I20260619 00:53:31.310949 7852 tablet_bootstrap.cc:654] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:31.312181 7852 log.cc:824] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:31.312032 7849 raft_consensus.cc:348] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.312489 7849 raft_consensus.cc:374] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:31.312593 7849 raft_consensus.cc:729] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb2aa30dbb54ca6a5a38eb997f4ac51, State: Initialized, Role: FOLLOWER
I20260619 00:53:31.313365 7849 consensus_queue.cc:260] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.315065 7852 tablet_bootstrap.cc:492] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: No bootstrap required, opened a new log
I20260619 00:53:31.315548 7852 ts_tablet_manager.cc:1397] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260619 00:53:31.317684 7850 raft_consensus.cc:348] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.318118 7850 raft_consensus.cc:374] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:31.318212 7850 raft_consensus.cc:729] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510b5fc1e9d94f40b358deb74cddd9b4, State: Initialized, Role: FOLLOWER
I20260619 00:53:31.318661 7849 ts_tablet_manager.cc:1428] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Time spent starting tablet: real 0.017s user 0.005s sys 0.011s
I20260619 00:53:31.319265 7850 consensus_queue.cc:260] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.325495 7852 raft_consensus.cc:348] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.326480 7852 raft_consensus.cc:374] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:31.326622 7852 raft_consensus.cc:729] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6807ccc12ab74177a2774fc89f23df1b, State: Initialized, Role: FOLLOWER
I20260619 00:53:31.326843 7850 ts_tablet_manager.cc:1428] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260619 00:53:31.326978 7814 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
I20260619 00:53:31.327390 7852 consensus_queue.cc:260] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.329613 7852 ts_tablet_manager.cc:1428] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 00:53:31.329948 7673 heartbeater.cc:503] Master 127.0.97.62:41499 was elected leader, sending a full tablet report...
W20260619 00:53:31.368731 7535 tablet.cc:2369] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:31.386533 7816 tablet.cc:2369] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:53:31.495939 7675 tablet.cc:2369] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:31.569968 7855 raft_consensus.cc:482] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:31.570243 7855 raft_consensus.cc:504] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.571947 7855 leader_election.cc:288] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347), 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:31.578898 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.572365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55411 (local address 127.0.97.2:40347)
0619 00:53:31.572585 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.572594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.573224 (+ 630us) server_negotiation.cc:407] Connection header received
0619 00:53:31.573390 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.573394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.573455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.573535 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:31.574077 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.575159 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.577177 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.577965 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.578120 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.578220 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.578501 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.578625 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:53:31.579317 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.572400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60881 (local address 127.0.97.3:43991)
0619 00:53:31.572762 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.572768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.573269 (+ 501us) server_negotiation.cc:407] Connection header received
0619 00:53:31.573686 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.573691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.573747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.573811 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:31.574520 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.575549 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.577269 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.578029 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.578169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.578277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.578553 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.578672 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260619 00:53:31.579528 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.572218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.97.1:55411)
0619 00:53:31.573081 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.573102 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.573267 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.573604 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.573608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.573632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:31.573922 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.573934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.575354 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.575357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.577036 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.577048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.578192 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.578197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.578346 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.578369 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.578476 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.578602 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"spinlock_wait_cycles":10752,"thread_start_us":397,"threads_started":1}
I20260619 00:53:31.579550 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.572360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43991 (local address 127.0.97.1:60881)
0619 00:53:31.573093 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:31.573112 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:31.573267 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:31.573874 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:31.573880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:31.573912 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:31.574306 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.574315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.575661 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.575667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.577115 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:31.577133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.578263 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.578269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:31.578406 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.578427 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 00:53:31.578530 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 00:53:31.578650 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"spinlock_wait_cycles":2048,"thread_start_us":416,"threads_started":1}
I20260619 00:53:31.580665 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ffc1c2c7a941f58f45d4b2899e30b5" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" is_pre_election: true
I20260619 00:53:31.580737 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ffc1c2c7a941f58f45d4b2899e30b5" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b" is_pre_election: true
I20260619 00:53:31.581280 7768 raft_consensus.cc:2454] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 0.
I20260619 00:53:31.581280 7627 raft_consensus.cc:2454] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 0.
I20260619 00:53:31.581907 7413 leader_election.cc:302] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510b5fc1e9d94f40b358deb74cddd9b4, bcb2aa30dbb54ca6a5a38eb997f4ac51; no voters:
I20260619 00:53:31.582290 7855 raft_consensus.cc:2781] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:31.582437 7855 raft_consensus.cc:482] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:31.582556 7855 raft_consensus.cc:3037] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:31.584658 7855 raft_consensus.cc:504] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.585330 7855 leader_election.cc:288] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 election: Requested vote from peers 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347), 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991)
I20260619 00:53:31.585779 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ffc1c2c7a941f58f45d4b2899e30b5" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b"
I20260619 00:53:31.585870 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ffc1c2c7a941f58f45d4b2899e30b5" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
I20260619 00:53:31.585975 7627 raft_consensus.cc:3037] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:31.586105 7768 raft_consensus.cc:3037] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:31.588328 7627 raft_consensus.cc:2454] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 1.
I20260619 00:53:31.588347 7768 raft_consensus.cc:2454] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 1.
I20260619 00:53:31.588739 7413 leader_election.cc:302] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510b5fc1e9d94f40b358deb74cddd9b4, bcb2aa30dbb54ca6a5a38eb997f4ac51; no voters:
I20260619 00:53:31.589087 7855 raft_consensus.cc:2781] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:31.589551 7855 raft_consensus.cc:686] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 LEADER]: Becoming Leader. State: Replica: bcb2aa30dbb54ca6a5a38eb997f4ac51, State: Running, Role: LEADER
I20260619 00:53:31.590174 7855 consensus_queue.cc:237] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } }
I20260619 00:53:31.595315 6550 catalog_manager.cc:5495] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } health_report { overall_health: HEALTHY } } }
I20260619 00:53:31.649080 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.644296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.0.97.1:36365)
0619 00:53:31.644475 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.644480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.644496 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:53:31.644580 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.644584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.644665 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.644829 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:31.645268 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.645969 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.647284 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.647664 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.647757 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.648142 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:31.648630 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:31.648754 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.648792 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.648879 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:53:31.656904 7768 raft_consensus.cc:1261] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Refusing update from remote peer bcb2aa30dbb54ca6a5a38eb997f4ac51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:31.656945 7627 raft_consensus.cc:1261] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Refusing update from remote peer bcb2aa30dbb54ca6a5a38eb997f4ac51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:31.657859 7860 consensus_queue.cc:1035] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:31.658221 7855 consensus_queue.cc:1035] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:31.671478 7864 mvcc.cc:205] Tried to move back new op lower bound from 7298377366137249792 to 7298377365876666368. Current Snapshot: MvccSnapshot[applied={T|T < 7298377366137249792}]
I20260619 00:53:31.675748 7865 mvcc.cc:205] Tried to move back new op lower bound from 7298377366137249792 to 7298377365876666368. Current Snapshot: MvccSnapshot[applied={T|T < 7298377366137249792}]
I20260619 00:53:31.698005 6550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:31.698335 6550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:31.705924 6550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3e506e1c3364b11af306c7f83c7186b: Sending DeleteTablet for 3 replicas of tablet 83ffc1c2c7a941f58f45d4b2899e30b5
I20260619 00:53:31.707458 7748 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42682
I20260619 00:53:31.707593 7607 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44946
I20260619 00:53:31.708168 7467 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51968
I20260619 00:53:31.708829 7466 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ffc1c2c7a941f58f45d4b2899e30b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51968
I20260619 00:53:31.719192 7876 tablet_replica.cc:330] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: stopping tablet replica
I20260619 00:53:31.719704 7876 raft_consensus.cc:2229] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:31.720503 7876 raft_consensus.cc:2258] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:31.721320 7874 tablet_replica.cc:330] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: stopping tablet replica
I20260619 00:53:31.721758 7874 raft_consensus.cc:2229] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:31.722198 7874 raft_consensus.cc:2258] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:31.724548 7876 ts_tablet_manager.cc:1905] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:31.725497 7875 tablet_replica.cc:330] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: stopping tablet replica
I20260619 00:53:31.728058 7875 raft_consensus.cc:2229] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:31.728576 7875 raft_consensus.cc:2258] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:31.728658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 6518
I20260619 00:53:31.731187 7875 ts_tablet_manager.cc:1905] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:31.731410 6580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:31.733759 7876 ts_tablet_manager.cc:1918] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:31.733948 7876 log.cc:1196] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/83ffc1c2c7a941f58f45d4b2899e30b5
I20260619 00:53:31.734465 7876 ts_tablet_manager.cc:1939] T 83ffc1c2c7a941f58f45d4b2899e30b5 P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting consensus metadata
I20260619 00:53:31.738186 7875 ts_tablet_manager.cc:1918] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:31.738368 7875 log.cc:1196] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/83ffc1c2c7a941f58f45d4b2899e30b5
I20260619 00:53:31.738822 7875 ts_tablet_manager.cc:1939] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 6807ccc12ab74177a2774fc89f23df1b: Deleting consensus metadata
I20260619 00:53:31.740760 7874 ts_tablet_manager.cc:1905] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:31.748181 7874 ts_tablet_manager.cc:1918] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:31.748407 7874 log.cc:1196] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/83ffc1c2c7a941f58f45d4b2899e30b5
I20260619 00:53:31.748999 7874 ts_tablet_manager.cc:1939] T 83ffc1c2c7a941f58f45d4b2899e30b5 P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting consensus metadata
I20260619 00:53:31.758082 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.710084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.97.61:39429)
0619 00:53:31.710306 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.710315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.727656 (+ 17341us) server_negotiation.cc:407] Connection header received
0619 00:53:31.737618 (+ 9962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.737625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.737723 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.737802 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.739894 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.741044 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.744953 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.745900 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.746060 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.751276 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.751281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.751285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.751325 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.754867 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.755865 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.755872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.755876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.756038 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.756496 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.756502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.756506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.756906 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.757149 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.757520 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.757757 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 00:53:31.775792 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.727905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.0.97.60:33489)
0619 00:53:31.747365 (+ 19460us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.747372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.758738 (+ 11366us) server_negotiation.cc:407] Connection header received
0619 00:53:31.758991 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.758998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.759123 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.759211 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.761305 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.762506 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.764528 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.765257 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.765389 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.768701 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.768708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.768715 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.768761 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.772518 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.773384 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.773390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.773394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.773545 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.774005 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.774010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.774014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.774397 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.774643 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.775173 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.775422 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19330,"thread_start_us":19469,"threads_started":1}
I20260619 00:53:31.870993 6537 catalog_manager.cc:4841] TS 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991): tablet 83ffc1c2c7a941f58f45d4b2899e30b5 (table test-table [id=31d312d25ab74be591d199bcc39b2f01]) successfully deleted
I20260619 00:53:31.872435 6534 catalog_manager.cc:4841] TS 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347): tablet 83ffc1c2c7a941f58f45d4b2899e30b5 (table test-table [id=31d312d25ab74be591d199bcc39b2f01]) successfully deleted
I20260619 00:53:31.872941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 6518
W20260619 00:53:31.902781 7410 connection.cc:535] server connection from 127.0.0.1:51968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 00:53:31.933279 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.907390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.97.61:39429)
0619 00:53:31.907627 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.907634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.914983 (+ 7349us) server_negotiation.cc:407] Connection header received
0619 00:53:31.915637 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.915647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.915763 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.915858 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.919120 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.920590 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.922580 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.923278 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.923399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.926668 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.926673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.926676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.926708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.930044 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.931044 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.931050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.931054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.931205 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.931639 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.931646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.931651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.932011 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.932288 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.932639 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.932891 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:53:31.935410 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:31.907660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.0.97.60:33489)
0619 00:53:31.907835 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:31.907841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:31.918805 (+ 10964us) server_negotiation.cc:407] Connection header received
0619 00:53:31.919046 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:31.919051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:31.919131 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:31.919200 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:31.922351 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.923183 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.925137 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:31.925852 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:31.925983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:31.929097 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:31.929103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:31.929109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:31.929147 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:31.932510 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.933321 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.933326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.933330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.933468 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:31.933880 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:31.933885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:31.933889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:31.934180 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:31.934346 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:31.934882 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 00:53:31.935102 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:53:32.687552 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:32.686387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:34981)
0619 00:53:32.687077 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:32.687304 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":342,"threads_started":1}
W20260619 00:53:32.688685 7814 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:32.689102 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:32.687721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:48719)
0619 00:53:32.688747 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:32.688917 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":778,"threads_started":1}
W20260619 00:53:32.689620 7673 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:32.708508 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:32.707773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:48689)
0619 00:53:32.708236 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:32.708351 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
W20260619 00:53:32.708947 7533 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41499 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:33.207286 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3e506e1c3364b11af306c7f83c7186b)
I20260619 00:53:33.207548 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3e506e1c3364b11af306c7f83c7186b)
I20260619 00:53:33.207578 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } }
I20260619 00:53:33.207791 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } }
I20260619 00:53:33.209415 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499), cd6d1e41c6a74837add03c2f4ae2b8ae (127.0.97.61:39429)
I20260619 00:53:33.215900 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.209945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36934)
0619 00:53:33.215350 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.215573 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5297,"thread_start_us":360,"threads_started":1}
I20260619 00:53:33.216344 7891 leader_election.cc:288] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499), 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60:33489)
W20260619 00:53:33.217255 7139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:33.218532 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.209244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36922)
0619 00:53:33.217988 (+ 8744us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.218199 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8546,"threads_started":1}
I20260619 00:53:33.220072 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.209300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33489 (local address 127.0.0.1:41948)
0619 00:53:33.210562 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.210586 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.210799 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.211442 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.211449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.211480 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.211833 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.211847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.213879 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.213887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.216593 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.216613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.218921 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.218930 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.219234 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.219287 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.219395 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.219588 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":6600,"threads_started":1}
W20260619 00:53:33.221093 7329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:33.221268 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.209503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.97.60:33489)
0619 00:53:33.210092 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.210101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.210679 (+ 578us) server_negotiation.cc:407] Connection header received
0619 00:53:33.210975 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.210983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.211074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.211153 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.212025 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.213654 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.216825 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.218490 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.218648 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.218790 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.220469 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.220674 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260619 00:53:33.221793 7355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9deb79c8cd5248d89cc594599d7686a8" is_pre_election: true
I20260619 00:53:33.222151 7355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd6d1e41c6a74837add03c2f4ae2b8ae in term 1.
I20260619 00:53:33.222870 7140 leader_election.cc:302] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9deb79c8cd5248d89cc594599d7686a8, cd6d1e41c6a74837add03c2f4ae2b8ae; no voters:
I20260619 00:53:33.223282 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 00:53:33.223454 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3e506e1c3364b11af306c7f83c7186b)
I20260619 00:53:33.223552 7891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:53:33.224359 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.223963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36972)
0619 00:53:33.224089 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.224193 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 00:53:33.224490 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.223860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36958)
0619 00:53:33.224100 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.224258 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 00:53:33.224623 7329 leader_election.cc:334] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
W20260619 00:53:33.224648 7139 leader_election.cc:334] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:33.227063 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } }
I20260619 00:53:33.227371 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.209261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39429 (local address 127.0.0.1:46206)
0619 00:53:33.218903 (+ 9642us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.218926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.219142 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.219909 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.219917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.219952 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.220332 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.220345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.222448 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.222452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.224597 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.224616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.226407 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.226411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.226569 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.226604 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.226710 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.226867 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9527,"mutex_wait_us":152,"thread_start_us":8230,"threads_started":1}
I20260619 00:53:33.227514 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.210607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.0.97.61:39429)
0619 00:53:33.217076 (+ 6469us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.217083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.219019 (+ 1936us) server_negotiation.cc:407] Connection header received
0619 00:53:33.219334 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.219342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.219434 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.219517 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.220977 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.222252 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.224803 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.226184 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.226330 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.226448 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.226767 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.227003 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6337,"thread_start_us":6420,"threads_started":1}
I20260619 00:53:33.228055 7891 leader_election.cc:288] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 election: Requested vote from peers f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499), 9deb79c8cd5248d89cc594599d7686a8 (127.0.97.60:33489)
I20260619 00:53:33.228334 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9deb79c8cd5248d89cc594599d7686a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" is_pre_election: true
I20260619 00:53:33.228574 7355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9deb79c8cd5248d89cc594599d7686a8"
I20260619 00:53:33.228735 7165 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9deb79c8cd5248d89cc594599d7686a8 in current term 2: Already voted for candidate cd6d1e41c6a74837add03c2f4ae2b8ae in this term.
I20260619 00:53:33.228844 7355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:53:33.229278 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.228863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36986)
0619 00:53:33.229020 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.229108 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 00:53:33.229504 7331 leader_election.cc:302] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9deb79c8cd5248d89cc594599d7686a8; no voters: cd6d1e41c6a74837add03c2f4ae2b8ae, f3e506e1c3364b11af306c7f83c7186b
I20260619 00:53:33.230934 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.230597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:36994)
0619 00:53:33.230743 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.230820 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 00:53:33.231117 7139 leader_election.cc:334] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499): Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
I20260619 00:53:33.231637 7355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd6d1e41c6a74837add03c2f4ae2b8ae in term 2.
I20260619 00:53:33.231832 7890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 00:53:33.232038 7140 leader_election.cc:302] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9deb79c8cd5248d89cc594599d7686a8, cd6d1e41c6a74837add03c2f4ae2b8ae; no voters: f3e506e1c3364b11af306c7f83c7186b
I20260619 00:53:33.232374 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 2 FOLLOWER]: Leader election won for term 2
I20260619 00:53:33.232867 7891 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [term 2 LEADER]: Becoming Leader. State: Replica: cd6d1e41c6a74837add03c2f4ae2b8ae, State: Running, Role: LEADER
I20260619 00:53:33.233246 7891 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } }
I20260619 00:53:33.234853 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6d1e41c6a74837add03c2f4ae2b8ae. Latest consensus state: current_term: 2 leader_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:33.235152 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:33.235791 7902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:33.239391 7902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:33.240170 7902 catalog_manager.cc:1259] Loaded cluster ID: 27d07f98f5a44951a0748fd462b1060e
I20260619 00:53:33.240270 7902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:33.241459 7902 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:33.242265 7902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Loaded TSK: 0
I20260619 00:53:33.243093 7902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:53:33.288798 7154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:33.295828 7154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:33.300215 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.299823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:37008)
0619 00:53:33.300002 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.300076 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 00:53:33.300259 7355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [term 2 FOLLOWER]: Refusing update from remote peer cd6d1e41c6a74837add03c2f4ae2b8ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 00:53:33.301066 7900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 00:53:33.301352 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.300979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:37020)
0619 00:53:33.301111 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.301216 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 00:53:33.301532 7139 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae -> Peer f3e506e1c3364b11af306c7f83c7186b (127.0.97.62:41499): Couldn't send request to peer f3e506e1c3364b11af306c7f83c7186b. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 00:53:33.303511 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6d1e41c6a74837add03c2f4ae2b8ae. Latest consensus state: current_term: 2 leader_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:33.303840 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:33.305469 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:33.305735 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:33.306059 7904 mvcc.cc:205] Tried to move back new op lower bound from 7298377372870701056 to 7298377372606971904. Current Snapshot: MvccSnapshot[applied={T|T < 7298377372870701056}]
I20260619 00:53:33.306690 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3e506e1c3364b11af306c7f83c7186b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41499 } } peers { permanent_uuid: "cd6d1e41c6a74837add03c2f4ae2b8ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39429 } attrs { promote: false } } peers { permanent_uuid: "9deb79c8cd5248d89cc594599d7686a8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33489 } attrs { promote: false } } }
I20260619 00:53:33.306957 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9deb79c8cd5248d89cc594599d7686a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:53:33.327008 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.315799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.0.97.3:43991)
0619 00:53:33.316228 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.316234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.316253 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:53:33.316332 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.316339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.316413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.316992 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.317187 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.318208 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.322862 (+ 4654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.324596 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.325146 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.325344 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.326544 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.326722 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":459,"threads_started":1}
I20260619 00:53:33.327134 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.313991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43991 (local address 127.0.0.1:55370)
0619 00:53:33.314150 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.314170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.314334 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.316564 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.316569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.316592 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.316849 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.316861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.318427 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.318435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.321132 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.321155 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.324879 (+ 3724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.324889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.326334 (+ 1445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.326391 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.326509 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.326713 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1077}
I20260619 00:53:33.327639 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.314418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36365 (local address 127.0.0.1:52444)
0619 00:53:33.314548 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.314564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.314655 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.315591 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.315597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.315642 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.315912 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.315927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.317659 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.317668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.322552 (+ 4884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.322575 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.324873 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.324884 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.325232 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.325302 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.325434 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.325667 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":2125}
I20260619 00:53:33.328717 7748 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7aca9687034d38b58b5ce49160f00a (DEFAULT_TABLE table=test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:33.328874 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.314779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.0.97.1:36365)
0619 00:53:33.315248 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.315254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.315273 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:53:33.315344 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.315348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.315419 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.315518 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.316095 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.317456 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.322777 (+ 5321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.324507 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.328001 (+ 3494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.328156 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.328228 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.328453 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":257,"threads_started":1}
I20260619 00:53:33.329226 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7aca9687034d38b58b5ce49160f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:33.329596 7467 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7aca9687034d38b58b5ce49160f00a (DEFAULT_TABLE table=test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:33.330008 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7aca9687034d38b58b5ce49160f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:33.337324 7914 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Bootstrap starting.
I20260619 00:53:33.338418 7913 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Bootstrap starting.
I20260619 00:53:33.339061 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.316134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.0.1:35492)
0619 00:53:33.326939 (+ 10805us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.326960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.327100 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.333491 (+ 6391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.333496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.333525 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.333855 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.333865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.334915 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.334922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.336654 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.336666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.338247 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.338256 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.338462 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.338505 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.338575 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.338711 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10644,"thread_start_us":11058,"threads_started":1}
I20260619 00:53:33.339553 7914 tablet_bootstrap.cc:654] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:33.339951 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.317860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.0.97.2:40347)
0619 00:53:33.333212 (+ 15352us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.333217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.333233 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:53:33.333290 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.333293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.333347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.333471 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.334012 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.334700 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.336806 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.338004 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.339322 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.339450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.339507 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.339652 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15255,"thread_start_us":15716,"threads_started":1}
I20260619 00:53:33.340107 7913 tablet_bootstrap.cc:654] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:33.340760 7607 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7aca9687034d38b58b5ce49160f00a (DEFAULT_TABLE table=test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:33.341336 7607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7aca9687034d38b58b5ce49160f00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:33.341805 7914 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: No bootstrap required, opened a new log
I20260619 00:53:33.341971 7914 ts_tablet_manager.cc:1397] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 00:53:33.342629 7914 raft_consensus.cc:348] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.342854 7914 raft_consensus.cc:374] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:33.342962 7914 raft_consensus.cc:729] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb2aa30dbb54ca6a5a38eb997f4ac51, State: Initialized, Role: FOLLOWER
I20260619 00:53:33.343201 7914 consensus_queue.cc:260] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.346691 7914 ts_tablet_manager.cc:1428] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 00:53:33.349046 7534 heartbeater.cc:503] Master 127.0.97.61:39429 was elected leader, sending a full tablet report...
I20260619 00:53:33.351720 7913 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: No bootstrap required, opened a new log
I20260619 00:53:33.351933 7913 ts_tablet_manager.cc:1397] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260619 00:53:33.353092 7913 raft_consensus.cc:348] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.353408 7913 raft_consensus.cc:374] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:33.353523 7913 raft_consensus.cc:729] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510b5fc1e9d94f40b358deb74cddd9b4, State: Initialized, Role: FOLLOWER
I20260619 00:53:33.353878 7913 consensus_queue.cc:260] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.355772 7913 ts_tablet_manager.cc:1428] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 00:53:33.356020 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.344292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:60023)
0619 00:53:33.354932 (+ 10640us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.355098 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10463,"thread_start_us":10284,"threads_started":1}
I20260619 00:53:33.356493 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.355510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:55235)
0619 00:53:33.356105 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.356221 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"threads_started":1}
I20260619 00:53:33.356669 7919 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Bootstrap starting.
I20260619 00:53:33.357343 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.356935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:41587)
0619 00:53:33.357133 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.357221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 00:53:33.358191 7815 heartbeater.cc:503] Master 127.0.97.61:39429 was elected leader, sending a full tablet report...
I20260619 00:53:33.358631 7919 tablet_bootstrap.cc:654] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:33.359362 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.358959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:33291)
0619 00:53:33.359141 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.359219 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 00:53:33.361397 7919 tablet_bootstrap.cc:492] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: No bootstrap required, opened a new log
I20260619 00:53:33.361539 7919 ts_tablet_manager.cc:1397] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 00:53:33.362211 7919 raft_consensus.cc:348] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.362397 7919 raft_consensus.cc:374] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:33.362454 7919 raft_consensus.cc:729] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6807ccc12ab74177a2774fc89f23df1b, State: Initialized, Role: FOLLOWER
I20260619 00:53:33.362686 7919 consensus_queue.cc:260] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.363468 7919 ts_tablet_manager.cc:1428] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:53:33.365453 7674 heartbeater.cc:503] Master 127.0.97.61:39429 was elected leader, sending a full tablet report...
I20260619 00:53:33.365928 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.363768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:47417)
0619 00:53:33.365409 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.365583 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1495,"thread_start_us":1189,"threads_started":1}
I20260619 00:53:33.367236 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.366503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:52801)
0619 00:53:33.367045 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.367126 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260619 00:53:33.432930 7920 raft_consensus.cc:482] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:33.433192 7920 raft_consensus.cc:504] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.435945 7920 leader_election.cc:288] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365), 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:33.439044 7917 raft_consensus.cc:482] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:33.439239 7917 raft_consensus.cc:504] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.439998 7917 leader_election.cc:288] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991), 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:33.440873 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" is_pre_election: true
I20260619 00:53:33.441133 7768 raft_consensus.cc:2454] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 0.
I20260619 00:53:33.441759 7413 leader_election.cc:302] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510b5fc1e9d94f40b358deb74cddd9b4, bcb2aa30dbb54ca6a5a38eb997f4ac51; no voters:
I20260619 00:53:33.442220 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.435429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51607 (local address 127.0.97.2:40347)
0619 00:53:33.435633 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.435640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.435695 (+ 55us) server_negotiation.cc:407] Connection header received
0619 00:53:33.435896 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.435904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.435988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.436076 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.436588 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.437681 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.439792 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.441012 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.441131 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.441260 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.441817 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.441950 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:53:33.442431 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b" is_pre_election: true
I20260619 00:53:33.442719 7626 raft_consensus.cc:2454] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in term 0.
I20260619 00:53:33.442806 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.435356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40347 (local address 127.0.97.3:51607)
0619 00:53:33.435608 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.435629 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.435778 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.436141 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.436146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.436192 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.436441 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.436451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.437839 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.437846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.439601 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.439617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.441427 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.441433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.441629 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.441661 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.442372 (+ 711us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.442529 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 00:53:33.443074 7917 raft_consensus.cc:2781] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:33.443185 7917 raft_consensus.cc:482] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:33.443274 7917 raft_consensus.cc:3037] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:33.444141 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b" is_pre_election: true
I20260619 00:53:33.444381 7626 raft_consensus.cc:2454] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 510b5fc1e9d94f40b358deb74cddd9b4 in term 0.
I20260619 00:53:33.444444 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.435597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50557 (local address 127.0.97.1:36365)
0619 00:53:33.435783 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.435789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.437234 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 00:53:33.437457 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.437461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.437517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.437608 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:33.438198 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.439246 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.441651 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.442944 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.443039 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.443139 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.444053 (+ 914us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.444196 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:53:33.444932 7701 leader_election.cc:302] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 510b5fc1e9d94f40b358deb74cddd9b4, 6807ccc12ab74177a2774fc89f23df1b; no voters:
I20260619 00:53:33.445211 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.435367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36365 (local address 127.0.97.3:50557)
0619 00:53:33.437118 (+ 1751us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.437135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:33.437273 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:33.437636 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:33.437640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:33.437664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:33.438036 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.438046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.439411 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.439417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.441468 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:33.441484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.443678 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.443683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:33.443877 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.443910 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:53:33.444586 (+ 676us) client_negotiation.cc:240] Negotiation successful
0619 00:53:33.444739 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"mutex_wait_us":433,"thread_start_us":1080,"threads_started":1}
I20260619 00:53:33.445439 7920 raft_consensus.cc:2781] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:33.445541 7920 raft_consensus.cc:482] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:33.445618 7920 raft_consensus.cc:3037] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:33.446280 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" is_pre_election: true
I20260619 00:53:33.447686 7917 raft_consensus.cc:504] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.448112 7487 raft_consensus.cc:2379] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 510b5fc1e9d94f40b358deb74cddd9b4 in current term 1: Already voted for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in this term.
I20260619 00:53:33.448544 7920 raft_consensus.cc:504] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.448877 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "510b5fc1e9d94f40b358deb74cddd9b4"
I20260619 00:53:33.449164 7768 raft_consensus.cc:2379] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in current term 1: Already voted for candidate 510b5fc1e9d94f40b358deb74cddd9b4 in this term.
I20260619 00:53:33.449576 7917 leader_election.cc:288] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 election: Requested vote from peers 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991), 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:33.449182 7920 leader_election.cc:288] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [CANDIDATE]: Term 1 election: Requested vote from peers bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365), 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347)
I20260619 00:53:33.449720 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b"
I20260619 00:53:33.449928 7626 raft_consensus.cc:3037] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:33.450202 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6807ccc12ab74177a2774fc89f23df1b"
I20260619 00:53:33.450629 7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7aca9687034d38b58b5ce49160f00a" candidate_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51"
I20260619 00:53:33.450802 7410 leader_election.cc:302] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcb2aa30dbb54ca6a5a38eb997f4ac51; no voters: 510b5fc1e9d94f40b358deb74cddd9b4, 6807ccc12ab74177a2774fc89f23df1b
I20260619 00:53:33.450934 7487 raft_consensus.cc:2379] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 510b5fc1e9d94f40b358deb74cddd9b4 in current term 1: Already voted for candidate bcb2aa30dbb54ca6a5a38eb997f4ac51 in this term.
I20260619 00:53:33.451135 7917 raft_consensus.cc:2726] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 00:53:33.452323 7626 raft_consensus.cc:2454] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 510b5fc1e9d94f40b358deb74cddd9b4 in term 1.
I20260619 00:53:33.452675 7701 leader_election.cc:302] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 510b5fc1e9d94f40b358deb74cddd9b4, 6807ccc12ab74177a2774fc89f23df1b; no voters: bcb2aa30dbb54ca6a5a38eb997f4ac51
I20260619 00:53:33.453016 7920 raft_consensus.cc:2781] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:33.453496 7920 raft_consensus.cc:686] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 LEADER]: Becoming Leader. State: Replica: 510b5fc1e9d94f40b358deb74cddd9b4, State: Running, Role: LEADER
I20260619 00:53:33.453919 7920 consensus_queue.cc:237] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } }
I20260619 00:53:33.454479 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.454066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:51691)
0619 00:53:33.454269 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.454362 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 00:53:33.457569 7154 catalog_manager.cc:5495] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "510b5fc1e9d94f40b358deb74cddd9b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 } health_report { overall_health: UNKNOWN } } }
I20260619 00:53:33.502722 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.497025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.97.3:43991)
0619 00:53:33.497216 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.497222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.497242 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:53:33.497430 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.497436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.497537 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.497700 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:33.498120 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.499217 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.500595 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.501132 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.501234 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.501660 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:33.502242 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:33.502360 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.502449 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.502515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:53:33.506989 7626 raft_consensus.cc:1261] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Refusing update from remote peer 510b5fc1e9d94f40b358deb74cddd9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:33.507341 7487 raft_consensus.cc:1261] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Refusing update from remote peer 510b5fc1e9d94f40b358deb74cddd9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:33.507706 7926 consensus_queue.cc:1035] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6807ccc12ab74177a2774fc89f23df1b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:33.508172 7920 consensus_queue.cc:1035] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcb2aa30dbb54ca6a5a38eb997f4ac51" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:33.511047 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.510246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:43237)
0619 00:53:33.510481 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.510627 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 00:53:33.511564 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.511046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:59521)
0619 00:53:33.511202 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.511325 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:53:33.518013 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.517438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.1:38353)
0619 00:53:33.517681 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.517797 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 00:53:33.519018 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.518498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:37875)
0619 00:53:33.518689 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.518804 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 00:53:33.524943 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.524457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.3:34601)
0619 00:53:33.524648 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.524751 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 00:53:33.526360 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.525807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.97.2:50511)
0619 00:53:33.526074 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.526161 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 00:53:33.537107 7155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:33.537384 7155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:33.543066 7155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd6d1e41c6a74837add03c2f4ae2b8ae: Sending DeleteTablet for 3 replicas of tablet 8c7aca9687034d38b58b5ce49160f00a
I20260619 00:53:33.544106 7748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7aca9687034d38b58b5ce49160f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55370
I20260619 00:53:33.544520 7467 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7aca9687034d38b58b5ce49160f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52444
I20260619 00:53:33.545179 7607 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7aca9687034d38b58b5ce49160f00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:35492
I20260619 00:53:33.545244 7939 tablet_replica.cc:330] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: stopping tablet replica
I20260619 00:53:33.545584 7939 raft_consensus.cc:2229] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:33.545989 7939 raft_consensus.cc:2258] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:33.546092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7317
I20260619 00:53:33.553406 7939 ts_tablet_manager.cc:1905] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:33.556834 7940 tablet_replica.cc:330] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: stopping tablet replica
I20260619 00:53:33.557123 7940 raft_consensus.cc:2229] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:33.557521 7940 raft_consensus.cc:2258] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:33.558251 7938 tablet_replica.cc:330] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: stopping tablet replica
I20260619 00:53:33.558454 7938 raft_consensus.cc:2229] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:33.559105 7938 raft_consensus.cc:2258] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:33.559572 7940 ts_tablet_manager.cc:1905] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:33.560232 7938 ts_tablet_manager.cc:1905] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:33.564322 7939 ts_tablet_manager.cc:1918] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:33.564567 7939 log.cc:1196] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/8c7aca9687034d38b58b5ce49160f00a
I20260619 00:53:33.565294 7939 ts_tablet_manager.cc:1939] T 8c7aca9687034d38b58b5ce49160f00a P bcb2aa30dbb54ca6a5a38eb997f4ac51: Deleting consensus metadata
I20260619 00:53:33.566797 7141 catalog_manager.cc:4841] TS bcb2aa30dbb54ca6a5a38eb997f4ac51 (127.0.97.1:36365): tablet 8c7aca9687034d38b58b5ce49160f00a (table test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]) successfully deleted
I20260619 00:53:33.567574 7938 ts_tablet_manager.cc:1918] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:33.567744 7938 log.cc:1196] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/8c7aca9687034d38b58b5ce49160f00a
I20260619 00:53:33.568216 7938 ts_tablet_manager.cc:1939] T 8c7aca9687034d38b58b5ce49160f00a P 510b5fc1e9d94f40b358deb74cddd9b4: Deleting consensus metadata
I20260619 00:53:33.569398 7142 catalog_manager.cc:4841] TS 510b5fc1e9d94f40b358deb74cddd9b4 (127.0.97.3:43991): tablet 8c7aca9687034d38b58b5ce49160f00a (table test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]) successfully deleted
I20260619 00:53:33.573067 7940 ts_tablet_manager.cc:1918] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:33.573233 7940 log.cc:1196] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/8c7aca9687034d38b58b5ce49160f00a
I20260619 00:53:33.573724 7940 ts_tablet_manager.cc:1939] T 8c7aca9687034d38b58b5ce49160f00a P 6807ccc12ab74177a2774fc89f23df1b: Deleting consensus metadata
I20260619 00:53:33.574751 7139 catalog_manager.cc:4841] TS 6807ccc12ab74177a2774fc89f23df1b (127.0.97.2:40347): tablet 8c7aca9687034d38b58b5ce49160f00a (table test-table [id=13ced2ee23d84ffe96d05cfea1c8b600]) successfully deleted
I20260619 00:53:33.579777 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.547051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.97.1:36365)
0619 00:53:33.547246 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.547254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.558583 (+ 11329us) server_negotiation.cc:407] Connection header received
0619 00:53:33.562080 (+ 3497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.562086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.562185 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.562282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:33.564351 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.565547 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.567408 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.568111 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.568252 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.572376 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:33.572398 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:33.572421 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:33.572468 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:33.576942 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.577582 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.577593 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.577597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.577775 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.578133 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.578138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.578143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.578636 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:33.578954 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.579133 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.579362 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:53:33.580693 7527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:33.708233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7317
I20260619 00:53:33.730891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7544
I20260619 00:53:33.747607 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.731807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.97.2:40347)
0619 00:53:33.731950 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.731954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.731971 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:33.732224 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.732229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.732295 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.732389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:33.734281 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.735070 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.737178 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.737580 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.737711 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.741336 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:33.741353 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:33.741369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:33.741423 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:33.744403 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.745489 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.745496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.745499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.745629 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.746047 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.746050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.746052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.746398 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:33.746613 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.746983 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.747181 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 00:53:33.748477 7667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:33.843657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7544
I20260619 00:53:33.861887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7685
I20260619 00:53:33.878165 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.862665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.0.97.3:43991)
0619 00:53:33.862871 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:33.862878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:33.862901 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:53:33.863009 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:33.863017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:33.863105 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:33.863216 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:33.864649 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.865868 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.867143 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:33.867805 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:33.867956 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:33.870557 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:33.870575 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:33.870605 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:33.870653 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:33.874917 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.875586 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.875604 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.875608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.875803 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:33.876262 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:33.876268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:33.876273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:33.876744 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:33.877261 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:33.877546 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 00:53:33.877803 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 00:53:33.878949 7808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:33.899487 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.898713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:37034)
0619 00:53:33.899186 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.899279 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":267,"threads_started":1}
I20260619 00:53:33.900786 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:33.900398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41499 (local address 127.0.0.1:37050)
0619 00:53:33.900541 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:33.900622 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 00:53:34.006862 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7685
I20260619 00:53:34.027794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7011
I20260619 00:53:34.028983 7185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:34.150772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7011
W20260619 00:53:34.172508 7330 connection.cc:535] server connection from 127.0.0.1:41948 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 00:53:34.172924 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7110
I20260619 00:53:34.173939 7375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:34.292724 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7110
2026-06-19T00:53:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:34 dist-test-slave-r7bh krb5kdc[7951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:34 dist-test-slave-r7bh krb5kdc[7951](info): set up 2 sockets
Jun 19 00:53:34 dist-test-slave-r7bh krb5kdc[7951](info): commencing operation
krb5kdc: starting...
W20260619 00:53:35.389809 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:35 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 00:53:35.612546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:35.936147 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:35.936460 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:35.936550 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:35.943909 7964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:35.944031 7964 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 00:53:35.944073 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:35.944128 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:35.944178 7964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:35.944240 7964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:35.954391 7964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39047
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:35.955647 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:35.957116 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:35.969558 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:35.969774 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:35.969558 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:35.970506 7964 server_base.cc:1034] running on GCE node
I20260619 00:53:35.971372 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:35.972934 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:35.974192 7964 hybrid_clock.cc:648] HybridClock initialized: now 1781830415974122 us; error 57 us; skew 500 ppm
I20260619 00:53:35.974599 7964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:35 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:35.978060 7964 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 00:53:35.979882 7964 webserver.cc:460] Webserver started at http://127.0.97.62:46371/ using document root <none> and password file <none>
I20260619 00:53:35.980698 7964 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:35.980895 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:35.981277 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:35.985046 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "a8aac1a32fcc4a37a4842af1127ed956"
format_stamp: "Formatted at 2026-06-19 00:53:35 on dist-test-slave-r7bh"
server_key: "f1a8fe5392c38c6cd21862a626e54c3d"
server_key_iv: "2c3c75fb7c3c4904bc4a49b3e0ce5e58"
server_key_version: "encryptionkey@0"
I20260619 00:53:35.986374 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8aac1a32fcc4a37a4842af1127ed956"
format_stamp: "Formatted at 2026-06-19 00:53:35 on dist-test-slave-r7bh"
server_key: "f1a8fe5392c38c6cd21862a626e54c3d"
server_key_iv: "2c3c75fb7c3c4904bc4a49b3e0ce5e58"
server_key_version: "encryptionkey@0"
I20260619 00:53:35.992719 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 00:53:35.997376 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:35.998996 7964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:53:35.999212 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "a8aac1a32fcc4a37a4842af1127ed956"
format_stamp: "Formatted at 2026-06-19 00:53:35 on dist-test-slave-r7bh"
server_key: "f1a8fe5392c38c6cd21862a626e54c3d"
server_key_iv: "2c3c75fb7c3c4904bc4a49b3e0ce5e58"
server_key_version: "encryptionkey@0"
I20260619 00:53:35.999472 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:36.020943 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:36.025708 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:36.025990 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:36.048362 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39047
I20260619 00:53:36.048368 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39047 every 8 connection(s)
I20260619 00:53:36.049801 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:36.053470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 7964
I20260619 00:53:36.053678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:36.053933 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:36.054481 388 external_mini_cluster.cc:1428] Setting key db82d479b8e9a646f832488c0ccf6617
I20260619 00:53:36.069089 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Bootstrap starting.
Jun 19 00:53:36 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:36.073206 8041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:36.074714 8041 log.cc:824] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:36.078521 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: No bootstrap required, opened a new log
I20260619 00:53:36.080194 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:36.056497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.0.97.62:39047)
0619 00:53:36.057373 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:36.057385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:36.057426 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:53:36.058968 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:36.058988 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:36.059365 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:36.059758 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:36.061427 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.062884 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.064786 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.065740 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.065891 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:36.070986 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:36.071001 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:36.071025 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:36.071081 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:36.075098 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.076167 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.076176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.076183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.076438 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.076861 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.076866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.076869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.077550 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:36.077851 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:36.078271 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 00:53:36.078713 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":417,"threads_started":1}
I20260619 00:53:36.087281 8041 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } }
I20260619 00:53:36.087652 8041 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:36.087733 8041 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8aac1a32fcc4a37a4842af1127ed956, State: Initialized, Role: FOLLOWER
I20260619 00:53:36.088248 8041 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } }
I20260619 00:53:36.088438 8041 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:36.088531 8041 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:36.088650 8041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:36.090821 8041 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } }
I20260619 00:53:36.091292 8041 leader_election.cc:302] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8aac1a32fcc4a37a4842af1127ed956; no voters:
I20260619 00:53:36.091943 8041 leader_election.cc:288] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:36.092077 8046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:36.092931 8046 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [term 1 LEADER]: Becoming Leader. State: Replica: a8aac1a32fcc4a37a4842af1127ed956, State: Running, Role: LEADER
I20260619 00:53:36.093492 8046 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } }
I20260619 00:53:36.094302 8041 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:36.096485 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8aac1a32fcc4a37a4842af1127ed956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } } }
I20260619 00:53:36.096387 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8aac1a32fcc4a37a4842af1127ed956. Latest consensus state: current_term: 1 leader_uuid: "a8aac1a32fcc4a37a4842af1127ed956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8aac1a32fcc4a37a4842af1127ed956" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39047 } } }
I20260619 00:53:36.096864 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:36.096902 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:36.099817 8054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:36.105563 8054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:36.114501 8054 catalog_manager.cc:1347] Generated new cluster ID: cd86551421a24fb1a649887c49b61a1d
I20260619 00:53:36.114636 8054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:36.126804 8054 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:36.128733 8054 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:36.136264 8054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Generated new TSK 0
I20260619 00:53:36.137162 8054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 00:53:36.200376 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39047
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:36.551448 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:36.551762 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:36.551863 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:36.559412 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:36.559540 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:36.559624 8067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:53:36.559701 8067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:36.569981 8067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:36.571317 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:36.572659 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:36.586858 8075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:36.586889 8072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:36.586958 8073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:36.588369 8067 server_base.cc:1034] running on GCE node
I20260619 00:53:36.589345 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:36.590338 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:36.591565 8067 hybrid_clock.cc:648] HybridClock initialized: now 1781830416591476 us; error 63 us; skew 500 ppm
I20260619 00:53:36.591876 8067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:36 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:36.595268 8067 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 00:53:36.596966 8067 webserver.cc:460] Webserver started at http://127.0.97.1:45353/ using document root <none> and password file <none>
I20260619 00:53:36.597699 8067 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:36.597882 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:36.598191 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:36.601408 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "786a9a27af0d44148b89377ae3f89a35"
format_stamp: "Formatted at 2026-06-19 00:53:36 on dist-test-slave-r7bh"
server_key: "0fadc681de5b08ec37aa81c5c6dc332e"
server_key_iv: "cd2baade98f4cea8a2ce7a594e2b3166"
server_key_version: "encryptionkey@0"
I20260619 00:53:36.602240 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "786a9a27af0d44148b89377ae3f89a35"
format_stamp: "Formatted at 2026-06-19 00:53:36 on dist-test-slave-r7bh"
server_key: "0fadc681de5b08ec37aa81c5c6dc332e"
server_key_iv: "cd2baade98f4cea8a2ce7a594e2b3166"
server_key_version: "encryptionkey@0"
I20260619 00:53:36.608201 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 00:53:36.612035 8082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:36.613754 8067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:36.613919 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "786a9a27af0d44148b89377ae3f89a35"
format_stamp: "Formatted at 2026-06-19 00:53:36 on dist-test-slave-r7bh"
server_key: "0fadc681de5b08ec37aa81c5c6dc332e"
server_key_iv: "cd2baade98f4cea8a2ce7a594e2b3166"
server_key_version: "encryptionkey@0"
I20260619 00:53:36.614187 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:36.639303 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:36.643858 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:36.644163 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:36.646844 8067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:36.646965 8067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:36.647118 8067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:36.647205 8067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:53:36 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:36.672084 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:36.647710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34863 (local address 127.0.97.62:39047)
0619 00:53:36.648172 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:36.648178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:36.649507 (+ 1329us) server_negotiation.cc:407] Connection header received
0619 00:53:36.651555 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:36.651561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:36.651664 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:36.651790 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:36.654585 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.655419 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.658703 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.659174 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.659280 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:36.665287 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:36.665293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:36.665296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:36.665334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:36.667882 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.669100 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.669107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.669113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.669270 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.669891 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.669895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.669897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.670277 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:36.670429 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:36.671473 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0619 00:53:36.671695 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260619 00:53:36.673722 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:36.648420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.1:34863)
0619 00:53:36.649333 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:36.649383 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:36.651273 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:36.652114 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:36.652130 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:36.653207 (+ 1077us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:36.654349 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:36.654390 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.655591 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.655599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:36.658511 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:36.658534 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.659406 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.659417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:36.659804 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:36.660866 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:36.660916 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:36.664971 (+ 4055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:36.668040 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:36.668053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:36.668077 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:36.668930 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:36.669394 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:36.669400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:36.669407 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:36.669725 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:36.670498 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:36.670505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:36.671073 (+ 568us) client_negotiation.cc:769] Sending connection context
0619 00:53:36.671377 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 00:53:36.671985 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":400,"threads_started":1}
I20260619 00:53:36.695386 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:36.679467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.1:47109)
0619 00:53:36.680134 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:36.680155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:36.680321 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:36.680848 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:36.680853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:36.681437 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:36.681719 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:36.681730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.683500 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.683504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:36.685084 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:36.685099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.686423 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.686435 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:36.686695 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:36.687412 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:36.687443 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:36.688717 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:36.692635 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:36.692646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:36.692650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:36.693163 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:36.693549 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:36.693554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:36.693557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:36.693696 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:36.694206 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:36.694211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:36.694468 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 00:53:36.694665 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 00:53:36.695065 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":385,"threads_started":1}
I20260619 00:53:36.695604 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:36.679380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47109 (local address 127.0.97.62:39047)
0619 00:53:36.679535 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:36.679539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:36.680252 (+ 713us) server_negotiation.cc:407] Connection header received
0619 00:53:36.680466 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:36.680470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:36.680531 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:36.680641 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:36.681883 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.683343 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.685415 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:36.686226 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:36.686377 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:36.688946 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:36.688957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:36.688965 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:36.689021 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:36.692506 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.693294 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.693300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.693305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.693438 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:36.693824 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:36.693835 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:36.693838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:36.694094 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:36.694252 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:36.695120 (+ 868us) server_negotiation.cc:299] Negotiation successful
0619 00:53:36.695280 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 00:53:36.703169 8067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40061
I20260619 00:53:36.703186 8202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40061 every 8 connection(s)
I20260619 00:53:36.704634 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:36.705988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8067
I20260619 00:53:36.706185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:36.706820 388 external_mini_cluster.cc:1428] Setting key 2587ecabf47122c61d80abefecf61904
I20260619 00:53:36.708542 8203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39047
I20260619 00:53:36.708806 8203 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:36.709621 8203 heartbeater.cc:511] Master 127.0.97.62:39047 requested a full tablet report, sending...
I20260619 00:53:36.711915 7996 ts_manager.cc:194] Registered new tserver with Master: 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061)
I20260619 00:53:36.714102 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34863
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 00:53:36.754390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39047
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:37.092252 8207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:37.092578 8207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:37.092763 8207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:37.100203 8207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:37.100360 8207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:37.100510 8207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:53:37.100651 8207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:37.110769 8207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:37.112109 8207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:37.113490 8207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:37.127184 8212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:37.127148 8215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:37.127117 8213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:37.128180 8207 server_base.cc:1034] running on GCE node
I20260619 00:53:37.128906 8207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:37.129854 8207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:37.131114 8207 hybrid_clock.cc:648] HybridClock initialized: now 1781830417131044 us; error 48 us; skew 500 ppm
I20260619 00:53:37.131448 8207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:37.134567 8207 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 00:53:37.135982 8207 webserver.cc:460] Webserver started at http://127.0.97.2:34437/ using document root <none> and password file <none>
I20260619 00:53:37.136885 8207 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:37.137008 8207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:37.137353 8207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:37.140206 8207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "5f80c908c21b4a74a93aaee6798b10d4"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "a20901cf8eea8acde32174cde9c28d74"
server_key_iv: "b29b81de2625df644f7d058263ec7440"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.141108 8207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5f80c908c21b4a74a93aaee6798b10d4"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "a20901cf8eea8acde32174cde9c28d74"
server_key_iv: "b29b81de2625df644f7d058263ec7440"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.146188 8207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 00:53:37.149782 8222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:37.151167 8207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:53:37.151361 8207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "5f80c908c21b4a74a93aaee6798b10d4"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "a20901cf8eea8acde32174cde9c28d74"
server_key_iv: "b29b81de2625df644f7d058263ec7440"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.151595 8207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:37.170456 8207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:37.174943 8207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:37.175278 8207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:37.178297 8207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:37.178421 8207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:37.178579 8207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:37.178665 8207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:37.202790 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.178785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38189 (local address 127.0.97.62:39047)
0619 00:53:37.179037 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.179046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.180582 (+ 1536us) server_negotiation.cc:407] Connection header received
0619 00:53:37.182849 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.182855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.182945 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.183074 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.185622 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.186720 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.189425 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.190033 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.190174 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.194917 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:37.194927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:37.194936 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:37.194996 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:37.199151 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.200227 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.200230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.200233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.200354 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.201005 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.201010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.201013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.201316 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:37.201478 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.202232 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.202447 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:53:37.203922 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.179476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.2:38189)
0619 00:53:37.180379 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.180434 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.182470 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.183382 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.183398 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.184267 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:37.185394 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.185429 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.186929 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.186933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.189256 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.189270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.190524 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.190535 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.190862 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.191730 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:37.191753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:37.194573 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:37.199332 (+ 4759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.199346 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.199396 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.200054 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.200531 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.200539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.200545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.200877 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.201446 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:37.201455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:37.201852 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.202113 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.202666 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":386,"threads_started":1}
I20260619 00:53:37.227253 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.209542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48271 (local address 127.0.97.62:39047)
0619 00:53:37.209741 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.209746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.211114 (+ 1368us) server_negotiation.cc:407] Connection header received
0619 00:53:37.211262 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.211270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.211371 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.211495 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.212972 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.214297 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.216010 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.216847 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.217662 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.220242 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:37.220251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:37.220257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:37.220310 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:37.223714 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.224837 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.224845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.224851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.225012 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.225577 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.225585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.225588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.225967 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:37.226075 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.226749 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.227020 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 00:53:37.227841 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.209965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.2:48271)
0619 00:53:37.210986 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.211006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.211133 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.211550 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.211557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.212363 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:37.212743 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.212754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.214499 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.214506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.215830 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.215845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.217146 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.217160 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.217468 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.218479 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:37.218510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:37.219993 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:37.223877 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.223894 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.223904 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.224663 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.225160 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.225168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.225173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.225374 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.226120 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:37.226129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:37.226389 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.226630 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.227119 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":600,"threads_started":1}
I20260619 00:53:37.233471 8207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38795
I20260619 00:53:37.233492 8341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38795 every 8 connection(s)
I20260619 00:53:37.234772 8207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:37.236331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8207
I20260619 00:53:37.236507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:37.237005 8342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39047
I20260619 00:53:37.237071 388 external_mini_cluster.cc:1428] Setting key 88232be5a4c0a0e7c90b5ee7c3e8a75e
I20260619 00:53:37.237250 8342 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:37.238015 8342 heartbeater.cc:511] Master 127.0.97.62:39047 requested a full tablet report, sending...
I20260619 00:53:37.239246 7996 ts_manager.cc:194] Registered new tserver with Master: 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795)
I20260619 00:53:37.240026 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38189
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 00:53:37.284149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39047
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:37.613029 8346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:37.613348 8346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:37.613505 8346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:37.621013 8346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:37.621160 8346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:37.621310 8346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:53:37.621443 8346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:37.631973 8346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:37.633807 8346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:37.635247 8346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:37.648747 8352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:37.648712 8354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:37.648787 8351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:37.649714 8346 server_base.cc:1034] running on GCE node
I20260619 00:53:37.650424 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:37.651476 8346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:37.652774 8346 hybrid_clock.cc:648] HybridClock initialized: now 1781830417652661 us; error 74 us; skew 500 ppm
I20260619 00:53:37.653216 8346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:37.656967 8346 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 00:53:37.658633 8346 webserver.cc:460] Webserver started at http://127.0.97.3:38557/ using document root <none> and password file <none>
I20260619 00:53:37.659380 8346 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:37.659520 8346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:37.659871 8346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:37.663082 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "626c4332fd3848a09b51175daef86eb6"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "29e5eca50d299e1a519d7146a903c59c"
server_key_iv: "44bc8cc786fcc7c97c741ad58df67cd8"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.663961 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "626c4332fd3848a09b51175daef86eb6"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "29e5eca50d299e1a519d7146a903c59c"
server_key_iv: "44bc8cc786fcc7c97c741ad58df67cd8"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.669755 8346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 00:53:37.673560 8361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:37.674985 8346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:37.675199 8346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "626c4332fd3848a09b51175daef86eb6"
format_stamp: "Formatted at 2026-06-19 00:53:37 on dist-test-slave-r7bh"
server_key: "29e5eca50d299e1a519d7146a903c59c"
server_key_iv: "44bc8cc786fcc7c97c741ad58df67cd8"
server_key_version: "encryptionkey@0"
I20260619 00:53:37.675460 8346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:37.697227 8346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:37.701797 8346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:37.702092 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:37.705142 8346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:37.705268 8346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:37.705461 8346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:37.705607 8346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 00:53:37.717626 8203 heartbeater.cc:503] Master 127.0.97.62:39047 was elected leader, sending a full tablet report...
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:37.734882 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.706450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52533 (local address 127.0.97.62:39047)
0619 00:53:37.706631 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.706637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.708056 (+ 1419us) server_negotiation.cc:407] Connection header received
0619 00:53:37.710994 (+ 2938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.711003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.711123 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.711252 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.713886 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.715017 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.718087 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.718953 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.719118 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.725550 (+ 6432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:37.725561 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:37.725569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:37.725629 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:37.729829 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.731192 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.731201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.731206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.731435 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.732107 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.732113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.732119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.732607 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:37.732780 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.733955 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.734262 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 00:53:37.736589 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.706947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.3:52533)
0619 00:53:37.707869 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.707926 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.710515 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.711586 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.711604 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.712568 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:37.713661 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.713698 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.715212 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.715221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.717887 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.717900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.719480 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.719494 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.719884 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.720790 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:37.720862 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:37.725150 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:37.730046 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.730060 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.730089 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.730990 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.731576 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.731587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.731593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.731951 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.732771 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:37.732785 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:37.733382 (+ 597us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.733786 (+ 404us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.734549 (+ 763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":365,"threads_started":1}
I20260619 00:53:37.746785 8346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44389
I20260619 00:53:37.746873 8480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44389 every 8 connection(s)
I20260619 00:53:37.748363 8346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:37.750720 8481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39047
I20260619 00:53:37.751032 8481 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:37.751744 8481 heartbeater.cc:511] Master 127.0.97.62:39047 requested a full tablet report, sending...
I20260619 00:53:37.753298 7996 ts_manager.cc:194] Registered new tserver with Master: 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:37.754500 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52533
I20260619 00:53:37.755476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8346
I20260619 00:53:37.755651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:37.756074 388 external_mini_cluster.cc:1428] Setting key 03cfc68f2703b4307bb75b6c8329efb6
I20260619 00:53:37.758972 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:53:37.759275 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.744726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44715 (local address 127.0.97.62:39047)
0619 00:53:37.744940 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.744944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.745675 (+ 731us) server_negotiation.cc:407] Connection header received
0619 00:53:37.745862 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.745871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.745974 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.746102 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.747430 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.748513 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.750349 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.751031 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.751163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.753021 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:37.753026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:37.753031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:37.753065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:37.756585 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.757383 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.757386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.757389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.757512 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.757909 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.757913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.757916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.758173 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:37.758342 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.758701 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.758914 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:53:37.759356 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.744694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.97.3:44715)
0619 00:53:37.745537 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.745561 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.745716 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.746145 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.746152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.746952 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:37.747283 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.747293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.748629 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.748632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.750187 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.750200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.751233 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.751238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.751388 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.752000 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:37.752017 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:37.752882 (+ 865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:37.756726 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.756731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.756736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.757243 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.757618 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:37.757624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:37.757628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:37.757807 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:37.758293 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:37.758298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:37.758468 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.758603 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.758914 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":493,"threads_started":1}
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 00:53:37.809911 8091 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:37.798607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.97.62:39047)
0619 00:53:37.798895 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.798905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.799178 (+ 273us) server_negotiation.cc:407] Connection header received
0619 00:53:37.799428 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.799433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.799506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.799641 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.801030 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.801815 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.803131 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.803558 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.803707 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.809622 (+ 5915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57224: BlockingRecv error: recv got EOF from 127.0.0.1:57224 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 00:53:37 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830417, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:37.827435 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.813546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.62:39047)
0619 00:53:37.813794 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.813801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.814035 (+ 234us) server_negotiation.cc:407] Connection header received
0619 00:53:37.814249 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.814255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.814342 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.814450 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:37.815622 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.816335 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.817727 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.818224 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.818385 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.822308 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:37.822314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:37.822317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:37.822361 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:37.824932 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.825575 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.825581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.825584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.825719 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:37.826078 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:37.826084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:37.826087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:37.826359 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:37.826647 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.826914 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.827123 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:53:37.831575 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:37.838332 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:37.870190 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.851486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.97.3:44389)
0619 00:53:37.852313 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.852323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.855829 (+ 3506us) server_negotiation.cc:407] Connection header received
0619 00:53:37.856182 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.856195 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.856585 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.856790 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:37.859774 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.862109 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.867123 (+ 5014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.868760 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.868931 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.869204 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.869626 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.869831 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":373,"threads_started":1}
I20260619 00:53:37.870231 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.850653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44389 (local address 127.0.0.1:33628)
0619 00:53:37.855687 (+ 5034us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.855708 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.855962 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.858851 (+ 2889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.858867 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.858914 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:37.859460 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.859477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.863886 (+ 4409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.863895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.866921 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.866934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.869064 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.869072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.869334 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.869396 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.869529 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.869722 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4874,"mutex_wait_us":76,"thread_start_us":2359,"threads_started":1}
I20260619 00:53:37.873632 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.853914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.0.1:48240)
0619 00:53:37.866377 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.866397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.866581 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.867257 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.867261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.867294 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:37.867905 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.867918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.869764 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.869769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.871434 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.871446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.872741 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.872747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.872942 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.872975 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.873097 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.873243 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12335,"thread_start_us":12169,"threads_started":1}
I20260619 00:53:37.873605 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.854416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.97.1:40061)
0619 00:53:37.865562 (+ 11146us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.865571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.866580 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 00:53:37.866760 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.866776 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.867048 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.867271 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:37.868131 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.869564 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.871590 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.872524 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.872700 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.872962 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.873147 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.873303 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10962,"thread_start_us":11144,"threads_started":1}
I20260619 00:53:37.874083 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.850649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38795 (local address 127.0.0.1:39924)
0619 00:53:37.853456 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.853502 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.854976 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.865346 (+ 10370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.865356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.865386 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:37.865886 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.865908 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.867987 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.867993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.870181 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.870196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.873303 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.873310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.873482 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.873513 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.873578 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.873712 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2559,"mutex_wait_us":1,"thread_start_us":4701,"threads_started":1}
I20260619 00:53:37.875216 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.850828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.97.2:38795)
0619 00:53:37.864153 (+ 13325us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.864169 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.864212 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:53:37.864350 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.864362 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.864754 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.866252 (+ 1498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:37.866460 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.867818 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.870376 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.873041 (+ 2665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.874446 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.874647 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.874728 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.874899 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13066,"thread_start_us":13990,"threads_started":1}
I20260619 00:53:37.874665 8411 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 (DEFAULT_TABLE table=test-table [id=804781d0c83f463c89459f9ef3b4b499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:37.876368 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2587ef98c04f8ba62c43b31dc07b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:37.877995 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 (DEFAULT_TABLE table=test-table [id=804781d0c83f463c89459f9ef3b4b499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:37.878732 8272 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 (DEFAULT_TABLE table=test-table [id=804781d0c83f463c89459f9ef3b4b499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:37.879462 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2587ef98c04f8ba62c43b31dc07b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:37.880102 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2587ef98c04f8ba62c43b31dc07b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:37.889097 8505 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Bootstrap starting.
I20260619 00:53:37.893069 8505 tablet_bootstrap.cc:654] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:37.894224 8505 log.cc:824] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:37.894743 8506 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Bootstrap starting.
I20260619 00:53:37.896359 8507 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Bootstrap starting.
I20260619 00:53:37.898712 8506 tablet_bootstrap.cc:654] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:37.899540 8505 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: No bootstrap required, opened a new log
I20260619 00:53:37.899580 8507 tablet_bootstrap.cc:654] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:37.899823 8505 ts_tablet_manager.cc:1397] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260619 00:53:37.899986 8506 log.cc:824] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:37.900707 8507 log.cc:824] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:37.903667 8506 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: No bootstrap required, opened a new log
I20260619 00:53:37.904023 8506 ts_tablet_manager.cc:1397] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 00:53:37.907680 8507 tablet_bootstrap.cc:492] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: No bootstrap required, opened a new log
I20260619 00:53:37.907986 8507 ts_tablet_manager.cc:1397] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 00:53:37.908553 8505 raft_consensus.cc:348] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.909001 8505 raft_consensus.cc:374] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:37.909103 8505 raft_consensus.cc:729] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Initialized, Role: FOLLOWER
I20260619 00:53:37.909722 8505 consensus_queue.cc:260] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.910987 8481 heartbeater.cc:503] Master 127.0.97.62:39047 was elected leader, sending a full tablet report...
I20260619 00:53:37.913057 8505 ts_tablet_manager.cc:1428] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260619 00:53:37.914722 8506 raft_consensus.cc:348] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.915285 8506 raft_consensus.cc:374] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:37.915431 8506 raft_consensus.cc:729] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Initialized, Role: FOLLOWER
I20260619 00:53:37.916114 8506 consensus_queue.cc:260] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.917920 8506 ts_tablet_manager.cc:1428] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 00:53:37.918498 8507 raft_consensus.cc:348] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.919003 8507 raft_consensus.cc:374] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:37.919121 8507 raft_consensus.cc:729] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f80c908c21b4a74a93aaee6798b10d4, State: Initialized, Role: FOLLOWER
I20260619 00:53:37.919755 8507 consensus_queue.cc:260] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.920792 8342 heartbeater.cc:503] Master 127.0.97.62:39047 was elected leader, sending a full tablet report...
I20260619 00:53:37.921360 8507 ts_tablet_manager.cc:1428] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
W20260619 00:53:37.957063 8204 tablet.cc:2369] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:37.966528 8511 raft_consensus.cc:482] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:37.966873 8511 raft_consensus.cc:504] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.968603 8511 leader_election.cc:288] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061)
I20260619 00:53:37.974257 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.969199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52109 (local address 127.0.97.1:40061)
0619 00:53:37.969395 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.969402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.969424 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:53:37.969580 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.969584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.969654 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.969756 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:37.970156 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.970869 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.972475 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.973315 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.973463 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.973567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.973874 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.974002 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:53:37.974301 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.968970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.97.3:52109)
0619 00:53:37.969239 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.969263 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.969413 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.969785 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.969789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.969813 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:37.970010 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.970018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.971014 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.971020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.972304 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.972315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.973487 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.973493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.973704 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.973729 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.973849 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.973990 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260619 00:53:37.975339 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.969300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60519 (local address 127.0.97.2:38795)
0619 00:53:37.969491 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:37.969499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:37.969896 (+ 397us) server_negotiation.cc:407] Connection header received
0619 00:53:37.970109 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:37.970114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:37.970180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:37.970277 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:37.970794 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.971521 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.973238 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.974017 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.974180 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.974294 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:37.974962 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 00:53:37.975103 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 00:53:37.975389 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2587ef98c04f8ba62c43b31dc07b60" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786a9a27af0d44148b89377ae3f89a35" is_pre_election: true
I20260619 00:53:37.975983 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:37.969010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38795 (local address 127.0.97.3:60519)
0619 00:53:37.969786 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:37.969801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:37.969967 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:37.970300 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:37.970304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:37.970326 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:37.970648 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.970657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.971666 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.971672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.973081 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:37.973092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:37.974561 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:37.974570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:37.974779 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:37.974815 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 00:53:37.975501 (+ 686us) client_negotiation.cc:240] Negotiation successful
0619 00:53:37.975663 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":499,"threads_started":1}
I20260619 00:53:37.976022 8158 raft_consensus.cc:2454] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 0.
I20260619 00:53:37.976774 8365 leader_election.cc:302] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626c4332fd3848a09b51175daef86eb6, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:37.977255 8511 raft_consensus.cc:2781] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:37.977159 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2587ef98c04f8ba62c43b31dc07b60" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4" is_pre_election: true
I20260619 00:53:37.977381 8511 raft_consensus.cc:482] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:37.977494 8511 raft_consensus.cc:3037] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:37.977660 8297 raft_consensus.cc:2454] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 0.
I20260619 00:53:37.980135 8511 raft_consensus.cc:504] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
I20260619 00:53:37.980854 8511 leader_election.cc:288] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061)
I20260619 00:53:37.981353 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2587ef98c04f8ba62c43b31dc07b60" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4"
I20260619 00:53:37.981532 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2587ef98c04f8ba62c43b31dc07b60" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786a9a27af0d44148b89377ae3f89a35"
I20260619 00:53:37.981580 8297 raft_consensus.cc:3037] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:37.981769 8158 raft_consensus.cc:3037] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:37.984059 8297 raft_consensus.cc:2454] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 1.
I20260619 00:53:37.984400 8158 raft_consensus.cc:2454] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 1.
I20260619 00:53:37.984764 8362 leader_election.cc:302] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 626c4332fd3848a09b51175daef86eb6; no voters:
I20260619 00:53:37.985312 8511 raft_consensus.cc:2781] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:37.985752 8511 raft_consensus.cc:686] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Running, Role: LEADER
I20260619 00:53:37.986327 8511 consensus_queue.cc:237] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } }
W20260619 00:53:37.986872 8343 tablet.cc:2369] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:37.991088 7995 catalog_manager.cc:5495] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626c4332fd3848a09b51175daef86eb6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: UNKNOWN } } }
W20260619 00:53:38.001238 8482 tablet.cc:2369] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:53:38.019308 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.013877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.97.3:44389)
0619 00:53:38.014065 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.014070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.014087 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:38.014177 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.014180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.014271 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.014401 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:38.014802 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.015512 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.017006 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.017506 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.017603 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.018069 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.018735 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.018841 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.018966 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.019029 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:53:38.027724 8158 raft_consensus.cc:1261] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Refusing update from remote peer 626c4332fd3848a09b51175daef86eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.027722 8297 raft_consensus.cc:1261] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Refusing update from remote peer 626c4332fd3848a09b51175daef86eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.028896 8515 consensus_queue.cc:1035] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.029246 8511 consensus_queue.cc:1035] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.060514 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:38.060846 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:38.064548 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Sending DeleteTablet for 3 replicas of tablet 7a2587ef98c04f8ba62c43b31dc07b60
I20260619 00:53:38.066048 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260619 00:53:38.066177 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260619 00:53:38.067102 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2587ef98c04f8ba62c43b31dc07b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48240
I20260619 00:53:38.068758 8528 tablet_replica.cc:330] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: stopping tablet replica
I20260619 00:53:38.068608 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:38.069358 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:38.069314 8528 raft_consensus.cc:2229] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:38.069945 8528 raft_consensus.cc:2258] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.072705 8530 tablet_replica.cc:330] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: stopping tablet replica
I20260619 00:53:38.073189 8530 raft_consensus.cc:2229] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:38.073712 8530 raft_consensus.cc:2258] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.074472 8529 tablet_replica.cc:330] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: stopping tablet replica
I20260619 00:53:38.074908 8529 raft_consensus.cc:2229] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:38.076292 8529 raft_consensus.cc:2258] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.076951 8528 ts_tablet_manager.cc:1905] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.078678 8529 ts_tablet_manager.cc:1905] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.080024 8272 tablet_service.cc:1467] Processing CreateTablet for tablet c70e2bffd90c4efdb328166ef84456cd (DEFAULT_TABLE table=test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.080585 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e2bffd90c4efdb328166ef84456cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.084486 8528 ts_tablet_manager.cc:1918] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:38.084769 8528 log.cc:1196] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/7a2587ef98c04f8ba62c43b31dc07b60
I20260619 00:53:38.085397 8528 ts_tablet_manager.cc:1939] T 7a2587ef98c04f8ba62c43b31dc07b60 P 5f80c908c21b4a74a93aaee6798b10d4: Deleting consensus metadata
I20260619 00:53:38.085559 8411 tablet_service.cc:1467] Processing CreateTablet for tablet c70e2bffd90c4efdb328166ef84456cd (DEFAULT_TABLE table=test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.086067 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e2bffd90c4efdb328166ef84456cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.086889 8529 ts_tablet_manager.cc:1918] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:38.087100 8529 log.cc:1196] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/7a2587ef98c04f8ba62c43b31dc07b60
I20260619 00:53:38.087656 7980 catalog_manager.cc:4841] TS 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795): tablet 7a2587ef98c04f8ba62c43b31dc07b60 (table test-table [id=804781d0c83f463c89459f9ef3b4b499]) successfully deleted
I20260619 00:53:38.087750 8529 ts_tablet_manager.cc:1939] T 7a2587ef98c04f8ba62c43b31dc07b60 P 626c4332fd3848a09b51175daef86eb6: Deleting consensus metadata
I20260619 00:53:38.088522 8133 tablet_service.cc:1467] Processing CreateTablet for tablet c70e2bffd90c4efdb328166ef84456cd (DEFAULT_TABLE table=test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.089071 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c70e2bffd90c4efdb328166ef84456cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.090349 8507 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Bootstrap starting.
I20260619 00:53:38.091980 8507 tablet_bootstrap.cc:654] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.092674 8530 ts_tablet_manager.cc:1905] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.096740 7983 catalog_manager.cc:4841] TS 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389): tablet 7a2587ef98c04f8ba62c43b31dc07b60 (table test-table [id=804781d0c83f463c89459f9ef3b4b499]) successfully deleted
I20260619 00:53:38.098711 8505 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Bootstrap starting.
I20260619 00:53:38.100049 8506 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Bootstrap starting.
I20260619 00:53:38.100644 8505 tablet_bootstrap.cc:654] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.102216 8506 tablet_bootstrap.cc:654] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.105863 8506 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: No bootstrap required, opened a new log
I20260619 00:53:38.106030 8506 ts_tablet_manager.cc:1397] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 00:53:38.106987 8506 raft_consensus.cc:348] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.107226 8506 raft_consensus.cc:374] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.107327 8506 raft_consensus.cc:729] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.107654 8506 consensus_queue.cc:260] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.108680 8530 ts_tablet_manager.cc:1918] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:38.108870 8530 log.cc:1196] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/7a2587ef98c04f8ba62c43b31dc07b60
I20260619 00:53:38.109455 8530 ts_tablet_manager.cc:1939] T 7a2587ef98c04f8ba62c43b31dc07b60 P 786a9a27af0d44148b89377ae3f89a35: Deleting consensus metadata
I20260619 00:53:38.109583 8505 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: No bootstrap required, opened a new log
I20260619 00:53:38.109807 8505 ts_tablet_manager.cc:1397] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 00:53:38.110378 8507 tablet_bootstrap.cc:492] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: No bootstrap required, opened a new log
I20260619 00:53:38.110598 8507 ts_tablet_manager.cc:1397] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.013s
I20260619 00:53:38.110646 8506 ts_tablet_manager.cc:1428] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 00:53:38.110977 8505 raft_consensus.cc:348] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.111282 8505 raft_consensus.cc:374] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.111439 8505 raft_consensus.cc:729] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.111523 7983 catalog_manager.cc:4841] TS 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061): tablet 7a2587ef98c04f8ba62c43b31dc07b60 (table test-table [id=804781d0c83f463c89459f9ef3b4b499]) successfully deleted
I20260619 00:53:38.111866 8505 consensus_queue.cc:260] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.112170 8507 raft_consensus.cc:348] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.112427 8507 raft_consensus.cc:374] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.112531 8507 raft_consensus.cc:729] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f80c908c21b4a74a93aaee6798b10d4, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.112793 8505 ts_tablet_manager.cc:1428] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:38.112882 8507 consensus_queue.cc:260] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.113484 8507 ts_tablet_manager.cc:1428] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:38.186276 8512 raft_consensus.cc:482] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:38.186467 8512 raft_consensus.cc:504] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.188088 8512 leader_election.cc:288] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:38.196456 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.188947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34341 (local address 127.0.97.2:38795)
0619 00:53:38.189142 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.189149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.189716 (+ 567us) server_negotiation.cc:407] Connection header received
0619 00:53:38.189934 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.189939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.190011 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.190103 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:38.190900 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.191952 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.193866 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.194862 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.195018 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.195116 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.195950 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.196108 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:53:38.196844 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.188586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55907 (local address 127.0.97.3:44389)
0619 00:53:38.188851 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.188860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.189430 (+ 570us) server_negotiation.cc:407] Connection header received
0619 00:53:38.189730 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.189737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.189840 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.189951 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:38.190614 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.191760 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.194428 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.195662 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.195787 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.195878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.196377 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.196509 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:53:38.197108 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.188402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38795 (local address 127.0.97.1:34341)
0619 00:53:38.189627 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.189644 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.189795 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.190166 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.190172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.190200 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:38.190702 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.190715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.192073 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.192078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.193712 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.193724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.195387 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.195397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.195681 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.195726 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.196554 (+ 828us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.196689 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":63,"thread_start_us":905,"threads_started":1}
I20260619 00:53:38.197613 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.188361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44389 (local address 127.0.97.1:55907)
0619 00:53:38.189305 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.189329 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.189549 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.189963 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.189967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.189988 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:38.190418 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.190432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.191915 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.191922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.194234 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.194250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.196021 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.196027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.196179 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.196218 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.197101 (+ 883us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.197253 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":1,"thread_start_us":650,"threads_started":1}
I20260619 00:53:38.197872 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e2bffd90c4efdb328166ef84456cd" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4" is_pre_election: true
I20260619 00:53:38.198148 8297 raft_consensus.cc:2454] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:38.198549 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e2bffd90c4efdb328166ef84456cd" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6" is_pre_election: true
I20260619 00:53:38.198690 8083 leader_election.cc:302] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:38.199007 8431 raft_consensus.cc:2454] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:38.199119 8512 raft_consensus.cc:2781] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:38.199242 8512 raft_consensus.cc:482] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:38.199345 8512 raft_consensus.cc:3037] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.201563 8512 raft_consensus.cc:504] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.202196 8512 leader_election.cc:288] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Requested vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:38.202647 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e2bffd90c4efdb328166ef84456cd" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4"
I20260619 00:53:38.202740 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c70e2bffd90c4efdb328166ef84456cd" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6"
I20260619 00:53:38.202885 8297 raft_consensus.cc:3037] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.202980 8431 raft_consensus.cc:3037] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.206048 8297 raft_consensus.cc:2454] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:38.206383 8431 raft_consensus.cc:2454] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:38.206568 8083 leader_election.cc:302] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:38.206945 8512 raft_consensus.cc:2781] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:38.207480 8512 raft_consensus.cc:686] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Becoming Leader. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Running, Role: LEADER
I20260619 00:53:38.208175 8512 consensus_queue.cc:237] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.212019 7995 catalog_manager.cc:5495] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "786a9a27af0d44148b89377ae3f89a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:53:38 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830415, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:38.267566 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.251888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.97.62:39047)
0619 00:53:38.252178 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.252187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.252805 (+ 618us) server_negotiation.cc:407] Connection header received
0619 00:53:38.253027 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.253033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.253140 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.253264 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:38.254411 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.255626 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.257150 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.257778 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.257921 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.261675 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:38.261683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:38.261690 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:38.261733 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:38.264383 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:38.265384 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:38.265388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:38.265390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:38.265514 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:38.265968 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:38.265972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:38.265974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:38.266241 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:38.266373 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.266980 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.267272 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":131}
I20260619 00:53:38.267813 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.251853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39047 (local address 127.0.0.1:57248)
0619 00:53:38.252680 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.252700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.252897 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.253306 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.253309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.253994 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:38.254256 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.254266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.255813 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.255817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.257005 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.257016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.257964 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.257969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.258081 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.258860 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:38.258884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:38.261443 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:38.264540 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:38.264557 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:38.264566 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:38.265242 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:38.265625 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:38.265630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:38.265635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:38.265817 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:38.266385 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:38.266394 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:38.266617 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.266807 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.267142 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":502,"threads_started":1}
I20260619 00:53:38.273293 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57248:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 00:53:38.274372 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:38.281581 8133 tablet_service.cc:1467] Processing CreateTablet for tablet ddefc0d944374a638a26ff364eadd95c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5bb36939a4241f8b78f59f06087b33f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:53:38.282207 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddefc0d944374a638a26ff364eadd95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.282120 8272 tablet_service.cc:1467] Processing CreateTablet for tablet ddefc0d944374a638a26ff364eadd95c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5bb36939a4241f8b78f59f06087b33f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:53:38.282689 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddefc0d944374a638a26ff364eadd95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.285780 8411 tablet_service.cc:1467] Processing CreateTablet for tablet ddefc0d944374a638a26ff364eadd95c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5bb36939a4241f8b78f59f06087b33f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:53:38.286376 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddefc0d944374a638a26ff364eadd95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.288959 8506 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: Bootstrap starting.
I20260619 00:53:38.290621 8506 tablet_bootstrap.cc:654] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.290683 8507 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: Bootstrap starting.
I20260619 00:53:38.292455 8507 tablet_bootstrap.cc:654] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.294864 8505 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: Bootstrap starting.
I20260619 00:53:38.297330 8505 tablet_bootstrap.cc:654] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.303131 8506 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: No bootstrap required, opened a new log
I20260619 00:53:38.303349 8506 ts_tablet_manager.cc:1397] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260619 00:53:38.304353 8506 raft_consensus.cc:348] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.304677 8506 raft_consensus.cc:374] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.304754 8506 raft_consensus.cc:729] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.305086 8506 consensus_queue.cc:260] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.305553 8512 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.305864 8512 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.306428 8506 ts_tablet_manager.cc:1428] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:38.305377 8507 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: No bootstrap required, opened a new log
I20260619 00:53:38.309152 8507 ts_tablet_manager.cc:1397] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.007s
I20260619 00:53:38.310027 8507 raft_consensus.cc:348] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.310272 8507 raft_consensus.cc:374] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.310362 8507 raft_consensus.cc:729] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f80c908c21b4a74a93aaee6798b10d4, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.310643 8507 consensus_queue.cc:260] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.310945 8505 tablet_bootstrap.cc:492] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: No bootstrap required, opened a new log
I20260619 00:53:38.311115 8505 ts_tablet_manager.cc:1397] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260619 00:53:38.311450 8507 ts_tablet_manager.cc:1428] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:53:38.311261 8513 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.311679 8513 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.311913 8505 raft_consensus.cc:348] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.312161 8505 raft_consensus.cc:374] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.312250 8505 raft_consensus.cc:729] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.312561 8505 consensus_queue.cc:260] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.313174 8505 ts_tablet_manager.cc:1428] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:53:38.313895 8515 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.314150 8515 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.549750 8515 raft_consensus.cc:482] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:38.550016 8515 raft_consensus.cc:504] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.550644 8515 leader_election.cc:288] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061), 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795)
I20260619 00:53:38.551296 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddefc0d944374a638a26ff364eadd95c" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4" is_pre_election: true
I20260619 00:53:38.551317 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddefc0d944374a638a26ff364eadd95c" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786a9a27af0d44148b89377ae3f89a35" is_pre_election: true
I20260619 00:53:38.551553 8158 raft_consensus.cc:2454] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 0.
I20260619 00:53:38.551564 8297 raft_consensus.cc:2454] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 0.
I20260619 00:53:38.552027 8365 leader_election.cc:302] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 626c4332fd3848a09b51175daef86eb6, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:38.552381 8515 raft_consensus.cc:2781] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:38.552479 8515 raft_consensus.cc:482] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:38.552640 8515 raft_consensus.cc:3037] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.554728 8515 raft_consensus.cc:504] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.555591 8515 leader_election.cc:288] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 election: Requested vote from peers 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061), 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795)
I20260619 00:53:38.556071 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddefc0d944374a638a26ff364eadd95c" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4"
I20260619 00:53:38.556118 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddefc0d944374a638a26ff364eadd95c" candidate_uuid: "626c4332fd3848a09b51175daef86eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786a9a27af0d44148b89377ae3f89a35"
I20260619 00:53:38.556360 8158 raft_consensus.cc:3037] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.556387 8297 raft_consensus.cc:3037] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:38.558391 8297 raft_consensus.cc:2454] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 1.
I20260619 00:53:38.558610 8158 raft_consensus.cc:2454] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626c4332fd3848a09b51175daef86eb6 in term 1.
I20260619 00:53:38.558873 8362 leader_election.cc:302] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 626c4332fd3848a09b51175daef86eb6; no voters:
I20260619 00:53:38.559185 8515 raft_consensus.cc:2781] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:38.559577 8515 raft_consensus.cc:686] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Running, Role: LEADER
I20260619 00:53:38.559888 8515 consensus_queue.cc:237] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.561595 8547 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: TxnStatusTablet state changed. Reason: New leader 626c4332fd3848a09b51175daef86eb6. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.561941 8547 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:53:38.562590 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 00:53:38.562866 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 00:53:38.563407 7996 catalog_manager.cc:5495] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626c4332fd3848a09b51175daef86eb6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } health_report { overall_health: HEALTHY } } }
I20260619 00:53:38.608268 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.602388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.0.97.3:44389)
0619 00:53:38.602569 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.602577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.602598 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:53:38.602892 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.602899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.602986 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.603091 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:38.603505 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.604294 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.606390 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.606833 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.607036 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.607509 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.607626 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.607721 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.607873 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.607954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:53:38.608309 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.602269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44389 (local address 127.0.0.1:33656)
0619 00:53:38.602518 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.602538 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.602687 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.603128 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.603132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.603145 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:53:38.603353 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.603362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.604470 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.604476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.606221 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.606236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.607110 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.607123 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.607258 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.607318 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.607725 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.607747 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.607854 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.608005 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:53:38.612628 8297 raft_consensus.cc:1261] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Refusing update from remote peer 626c4332fd3848a09b51175daef86eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.612870 8158 raft_consensus.cc:1261] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Refusing update from remote peer 626c4332fd3848a09b51175daef86eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.613312 8515 consensus_queue.cc:1035] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.613637 8547 consensus_queue.cc:1035] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.629837 8512 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: TxnStatusTablet state changed. Reason: New leader 626c4332fd3848a09b51175daef86eb6. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.630142 8512 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.631724 8513 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: TxnStatusTablet state changed. Reason: New leader 626c4332fd3848a09b51175daef86eb6. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.632053 8513 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.632424 8547 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.632696 8547 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:53:38.646483 8513 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.646755 8513 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 5f80c908c21b4a74a93aaee6798b10d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.650250 8547 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.650544 8547 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 626c4332fd3848a09b51175daef86eb6: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:53:38.653618 8512 tablet_replica.cc:439] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "626c4332fd3848a09b51175daef86eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } } }
I20260619 00:53:38.654000 8512 tablet_replica.cc:442] T ddefc0d944374a638a26ff364eadd95c P 786a9a27af0d44148b89377ae3f89a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:53:38.656373 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.645128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.97.1:40061)
0619 00:53:38.645892 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.645901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.649264 (+ 3363us) server_negotiation.cc:407] Connection header received
0619 00:53:38.649516 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.649523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.649651 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.649842 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:38.650342 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.651251 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.653152 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.653927 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.654182 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.654794 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.655670 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.655850 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.655906 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.655997 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":474,"threads_started":1}
I20260619 00:53:38.673054 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.666631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44389 (local address 127.0.97.1:47749)
0619 00:53:38.667187 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.667203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.667376 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.667891 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.667894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.667904 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:53:38.668171 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.668179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.669707 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.669711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.670826 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.670838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.671885 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.671889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.672010 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.672070 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.672513 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.672523 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.672591 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.672698 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260619 00:53:38.673076 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.666748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47749 (local address 127.0.97.3:44389)
0619 00:53:38.666972 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.666978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.667293 (+ 315us) server_negotiation.cc:407] Connection header received
0619 00:53:38.667629 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.667636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.667735 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.667857 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:38.668327 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.669552 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.670969 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.671690 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.671839 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.672251 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.672410 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.672510 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.672636 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.672721 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:53:38.683019 8297 raft_consensus.cc:1261] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.683248 8431 raft_consensus.cc:1261] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:38.683761 8536 consensus_queue.cc:1035] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.684491 8512 consensus_queue.cc:1035] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:38.692524 8553 mvcc.cc:205] Tried to move back new op lower bound from 7298377394919292928 to 7298377392984903680. Current Snapshot: MvccSnapshot[applied={T|T < 7298377394919292928}]
I20260619 00:53:38.705965 8552 mvcc.cc:205] Tried to move back new op lower bound from 7298377394919292928 to 7298377392984903680. Current Snapshot: MvccSnapshot[applied={T|T < 7298377394919292928}]
I20260619 00:53:38.743007 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.733030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44517 (local address 127.0.97.1:40061)
0619 00:53:38.733234 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.733243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.736120 (+ 2877us) server_negotiation.cc:407] Connection header received
0619 00:53:38.736396 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.736401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.736487 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.736588 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:38.737227 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.738383 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.740681 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.741413 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.741508 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.742195 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.742330 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.742388 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.742602 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.742688 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 00:53:38.743906 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.732995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.97.3:44517)
0619 00:53:38.736013 (+ 3018us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:38.736035 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:38.736239 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:38.736623 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:38.736629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:38.736642 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:53:38.737047 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.737060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.738556 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.738562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.740498 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:38.740514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.741704 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.741710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:38.741911 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.741992 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:53:38.742464 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:53:38.742475 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:53:38.743312 (+ 837us) client_negotiation.cc:240] Negotiation successful
0619 00:53:38.743492 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2789,"thread_start_us":2599,"threads_started":1}
I20260619 00:53:38.798265 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:38.798553 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:38.801919 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Sending DeleteTablet for 3 replicas of tablet c70e2bffd90c4efdb328166ef84456cd
I20260619 00:53:38.802953 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet c70e2bffd90c4efdb328166ef84456cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260619 00:53:38.803345 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet c70e2bffd90c4efdb328166ef84456cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48240
I20260619 00:53:38.804116 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet c70e2bffd90c4efdb328166ef84456cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260619 00:53:38.833295 8575 tablet_replica.cc:330] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: stopping tablet replica
I20260619 00:53:38.833537 8575 raft_consensus.cc:2229] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:38.833863 8575 raft_consensus.cc:2258] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.835090 8575 ts_tablet_manager.cc:1905] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.837461 8574 tablet_replica.cc:330] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: stopping tablet replica
I20260619 00:53:38.837666 8574 raft_consensus.cc:2229] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:38.838526 8573 tablet_replica.cc:330] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: stopping tablet replica
I20260619 00:53:38.838718 8573 raft_consensus.cc:2229] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:38.838959 8573 raft_consensus.cc:2258] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.839243 8574 raft_consensus.cc:2258] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:38.840173 8573 ts_tablet_manager.cc:1905] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.840562 8574 ts_tablet_manager.cc:1905] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:38.847345 8573 ts_tablet_manager.cc:1918] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:38.847489 8573 log.cc:1196] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/c70e2bffd90c4efdb328166ef84456cd
I20260619 00:53:38.847908 8573 ts_tablet_manager.cc:1939] T c70e2bffd90c4efdb328166ef84456cd P 5f80c908c21b4a74a93aaee6798b10d4: Deleting consensus metadata
I20260619 00:53:38.848843 8574 ts_tablet_manager.cc:1918] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:38.848980 7980 catalog_manager.cc:4841] TS 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795): tablet c70e2bffd90c4efdb328166ef84456cd (table test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]) successfully deleted
I20260619 00:53:38.849072 8574 log.cc:1196] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/c70e2bffd90c4efdb328166ef84456cd
I20260619 00:53:38.849619 8574 ts_tablet_manager.cc:1939] T c70e2bffd90c4efdb328166ef84456cd P 786a9a27af0d44148b89377ae3f89a35: Deleting consensus metadata
I20260619 00:53:38.850769 7983 catalog_manager.cc:4841] TS 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061): tablet c70e2bffd90c4efdb328166ef84456cd (table test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]) successfully deleted
I20260619 00:53:38.857295 8575 ts_tablet_manager.cc:1918] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:38.857532 8575 log.cc:1196] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/c70e2bffd90c4efdb328166ef84456cd
I20260619 00:53:38.858119 8575 ts_tablet_manager.cc:1939] T c70e2bffd90c4efdb328166ef84456cd P 626c4332fd3848a09b51175daef86eb6: Deleting consensus metadata
I20260619 00:53:38.859388 7983 catalog_manager.cc:4841] TS 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389): tablet c70e2bffd90c4efdb328166ef84456cd (table test-table [id=6dbc3e1e39c348d68c13e4e7348ce7ad]) successfully deleted
Jun 19 00:53:38 dist-test-slave-r7bh krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:53:38 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:38 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 00:53:38.903568 8582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:38.880048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.97.62:39047)
0619 00:53:38.887723 (+ 7675us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.887728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.887749 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:53:38.887807 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.887812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.887878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.888033 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:38.889599 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.890469 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.891819 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.892276 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.892429 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.903126 (+ 10697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57262: BlockingRecv error: recv got EOF from 127.0.0.1:57262 (error 108)
Metrics: {"server-negotiator.queue_time_us":7556,"thread_start_us":7429,"threads_started":1}
Jun 19 00:53:38 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 00:53:38.923033 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:38.906205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.97.62:39047)
0619 00:53:38.906379 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:38.906388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:38.909356 (+ 2968us) server_negotiation.cc:407] Connection header received
0619 00:53:38.909560 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:38.909572 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:38.909647 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:38.909755 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:38.910951 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.911680 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.912969 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:38.913383 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:38.913515 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:38.917366 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:38.917371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:38.917374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:38.917409 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:38.920200 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:38.920967 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:38.920972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:38.920977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:38.921141 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:38.921525 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:38.921530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:38.921534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:38.921886 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:38.922133 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:38.922438 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 00:53:38.922660 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:53:38.926683 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:53:38.927343 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:38.934042 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 116d740b653341a8a4ccaad80e9c957a (DEFAULT_TABLE table=test-table [id=baba0498a4a74dbb871a0f22350988d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.934345 8272 tablet_service.cc:1467] Processing CreateTablet for tablet 116d740b653341a8a4ccaad80e9c957a (DEFAULT_TABLE table=test-table [id=baba0498a4a74dbb871a0f22350988d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.934543 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116d740b653341a8a4ccaad80e9c957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.934723 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116d740b653341a8a4ccaad80e9c957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.939900 8411 tablet_service.cc:1467] Processing CreateTablet for tablet 116d740b653341a8a4ccaad80e9c957a (DEFAULT_TABLE table=test-table [id=baba0498a4a74dbb871a0f22350988d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:38.940364 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 116d740b653341a8a4ccaad80e9c957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:38.961194 8591 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Bootstrap starting.
I20260619 00:53:38.963862 8590 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Bootstrap starting.
I20260619 00:53:38.967027 8590 tablet_bootstrap.cc:654] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.968084 8591 tablet_bootstrap.cc:654] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.973457 8591 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: No bootstrap required, opened a new log
I20260619 00:53:38.973762 8591 ts_tablet_manager.cc:1397] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20260619 00:53:38.974737 8591 raft_consensus.cc:348] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.975759 8589 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Bootstrap starting.
I20260619 00:53:38.976943 8591 raft_consensus.cc:374] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.977074 8591 raft_consensus.cc:729] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f80c908c21b4a74a93aaee6798b10d4, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.977381 8591 consensus_queue.cc:260] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.977999 8591 ts_tablet_manager.cc:1428] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 00:53:38.978312 8589 tablet_bootstrap.cc:654] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:38.980226 8590 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: No bootstrap required, opened a new log
I20260619 00:53:38.980444 8590 ts_tablet_manager.cc:1397] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260619 00:53:38.983680 8590 raft_consensus.cc:348] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.984062 8590 raft_consensus.cc:374] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.984179 8590 raft_consensus.cc:729] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.984645 8590 consensus_queue.cc:260] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.985558 8590 ts_tablet_manager.cc:1428] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 00:53:38.990664 8589 tablet_bootstrap.cc:492] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: No bootstrap required, opened a new log
I20260619 00:53:38.990968 8589 ts_tablet_manager.cc:1397] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260619 00:53:38.992379 8589 raft_consensus.cc:348] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.992777 8589 raft_consensus.cc:374] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:38.992964 8589 raft_consensus.cc:729] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Initialized, Role: FOLLOWER
I20260619 00:53:38.993273 8589 consensus_queue.cc:260] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:38.994372 8589 ts_tablet_manager.cc:1428] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:39.050530 8513 raft_consensus.cc:482] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:39.050828 8513 raft_consensus.cc:504] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.052734 8513 leader_election.cc:288] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:39.059257 8512 raft_consensus.cc:482] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:39.059491 8512 raft_consensus.cc:504] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.060480 8512 leader_election.cc:288] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:39.060956 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4" is_pre_election: true
I20260619 00:53:39.061194 8297 raft_consensus.cc:2454] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:39.061331 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6" is_pre_election: true
I20260619 00:53:39.061561 8431 raft_consensus.cc:2454] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:39.061697 8083 leader_election.cc:302] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:39.061983 8512 raft_consensus.cc:2781] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:39.062110 8512 raft_consensus.cc:482] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:39.062215 8512 raft_consensus.cc:3037] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.065572 8512 raft_consensus.cc:504] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.066743 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4"
I20260619 00:53:39.066967 8297 raft_consensus.cc:3037] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.070046 8297 raft_consensus.cc:2454] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:39.070539 8083 leader_election.cc:302] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:39.070868 8536 raft_consensus.cc:2781] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:39.074469 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6"
I20260619 00:53:39.074759 8431 raft_consensus.cc:3037] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.076785 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.053602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57039 (local address 127.0.97.1:40061)
0619 00:53:39.053799 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:39.053805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:39.069424 (+ 15619us) server_negotiation.cc:407] Connection header received
0619 00:53:39.069716 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:39.069722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:39.069807 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:39.069872 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:39.070669 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.071873 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.074266 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.075570 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.075677 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.075794 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:39.076336 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 00:53:39.076488 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:53:39.077539 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.053231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40061 (local address 127.0.97.2:57039)
0619 00:53:39.069274 (+ 16043us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:39.069311 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:39.069548 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:39.070078 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:39.070083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:39.070110 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:39.070460 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:39.070473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.072053 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.072058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:39.074081 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:39.074097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.075952 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.075959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:39.076135 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.076171 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 00:53:39.077046 (+ 875us) client_negotiation.cc:240] Negotiation successful
0619 00:53:39.077197 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15889,"threads_started":1}
I20260619 00:53:39.077941 8431 raft_consensus.cc:2454] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:39.078680 8512 leader_election.cc:288] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Requested vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:39.082312 8157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "5f80c908c21b4a74a93aaee6798b10d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "786a9a27af0d44148b89377ae3f89a35" is_pre_election: true
I20260619 00:53:39.088007 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.079831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52945 (local address 127.0.97.3:44389)
0619 00:53:39.080001 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:39.080007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:39.081045 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 00:53:39.081320 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:39.081327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:39.081403 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:39.081464 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:53:39.082039 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.083181 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.085448 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.086785 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.086883 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.087001 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:39.087579 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 00:53:39.087723 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:53:39.088686 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.080759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44389 (local address 127.0.97.2:52945)
0619 00:53:39.080954 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:39.080974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:39.081157 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:39.081668 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:39.081674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:39.081702 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:53:39.081885 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:39.081896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.083381 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.083386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:39.085254 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:39.085271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.087135 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.087145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:39.087361 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.087426 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 00:53:39.088257 (+ 831us) client_negotiation.cc:240] Negotiation successful
0619 00:53:39.088416 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 00:53:39.089377 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "116d740b653341a8a4ccaad80e9c957a" candidate_uuid: "5f80c908c21b4a74a93aaee6798b10d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6" is_pre_election: true
I20260619 00:53:39.089747 8431 raft_consensus.cc:2379] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f80c908c21b4a74a93aaee6798b10d4 in current term 1: Already voted for candidate 786a9a27af0d44148b89377ae3f89a35 in this term.
I20260619 00:53:39.091408 8536 raft_consensus.cc:686] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Becoming Leader. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Running, Role: LEADER
I20260619 00:53:39.091732 8536 consensus_queue.cc:237] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.101449 8226 leader_election.cc:302] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4; no voters: 626c4332fd3848a09b51175daef86eb6, 786a9a27af0d44148b89377ae3f89a35
I20260619 00:53:39.102126 8513 raft_consensus.cc:2726] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 00:53:39.103152 7996 catalog_manager.cc:5495] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "786a9a27af0d44148b89377ae3f89a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20260619 00:53:39.124402 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.118745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.0.97.1:40061)
0619 00:53:39.118923 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:39.118929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:39.118974 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:53:39.119165 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:39.119171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:39.119257 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:39.119367 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:53:39.119730 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.120894 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.122297 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.123012 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.123148 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.123550 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:53:39.123685 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:53:39.123785 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:39.123969 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 00:53:39.124062 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 00:53:39.127211 8431 raft_consensus.cc:1261] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:39.127887 8297 raft_consensus.cc:1261] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:39.127877 8536 consensus_queue.cc:1035] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:39.128479 8536 consensus_queue.cc:1035] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:39.133927 8551 mvcc.cc:205] Tried to move back new op lower bound from 7298377396739383296 to 7298377396602789888. Current Snapshot: MvccSnapshot[applied={T|T < 7298377396739383296}]
I20260619 00:53:39.141141 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:39.141343 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:39.145243 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Sending DeleteTablet for 3 replicas of tablet 116d740b653341a8a4ccaad80e9c957a
I20260619 00:53:39.146281 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet 116d740b653341a8a4ccaad80e9c957a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260619 00:53:39.146549 8573 tablet_replica.cc:330] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: stopping tablet replica
I20260619 00:53:39.146765 8573 raft_consensus.cc:2229] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:39.146996 8573 raft_consensus.cc:2258] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.147838 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet 116d740b653341a8a4ccaad80e9c957a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260619 00:53:39.147905 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 00:53:39.148193 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet 116d740b653341a8a4ccaad80e9c957a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48240
W20260619 00:53:39.148626 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:53:39.148965 8575 tablet_replica.cc:330] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: stopping tablet replica
I20260619 00:53:39.149152 8575 raft_consensus.cc:2229] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:39.149194 8573 ts_tablet_manager.cc:1905] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.149407 8575 raft_consensus.cc:2258] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.149686 8574 tablet_replica.cc:330] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: stopping tablet replica
I20260619 00:53:39.149880 8574 raft_consensus.cc:2229] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:39.150185 8574 raft_consensus.cc:2258] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.153293 8575 ts_tablet_manager.cc:1905] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.156531 8573 ts_tablet_manager.cc:1918] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:39.156677 8573 log.cc:1196] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/116d740b653341a8a4ccaad80e9c957a
I20260619 00:53:39.157128 8573 ts_tablet_manager.cc:1939] T 116d740b653341a8a4ccaad80e9c957a P 5f80c908c21b4a74a93aaee6798b10d4: Deleting consensus metadata
I20260619 00:53:39.157658 8574 ts_tablet_manager.cc:1905] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.158309 7980 catalog_manager.cc:4841] TS 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795): tablet 116d740b653341a8a4ccaad80e9c957a (table test-table [id=baba0498a4a74dbb871a0f22350988d6]) successfully deleted
I20260619 00:53:39.164695 8575 ts_tablet_manager.cc:1918] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:39.164994 8575 log.cc:1196] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/116d740b653341a8a4ccaad80e9c957a
I20260619 00:53:39.165525 8575 ts_tablet_manager.cc:1939] T 116d740b653341a8a4ccaad80e9c957a P 626c4332fd3848a09b51175daef86eb6: Deleting consensus metadata
I20260619 00:53:39.165995 8574 ts_tablet_manager.cc:1918] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:53:39.166170 8574 log.cc:1196] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/116d740b653341a8a4ccaad80e9c957a
I20260619 00:53:39.166471 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee (DEFAULT_TABLE table=test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:39.165953 8411 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee (DEFAULT_TABLE table=test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:39.166887 8574 ts_tablet_manager.cc:1939] T 116d740b653341a8a4ccaad80e9c957a P 786a9a27af0d44148b89377ae3f89a35: Deleting consensus metadata
I20260619 00:53:39.166915 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb4d3f3f1b0416ebb7568c15e670cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:39.167022 8272 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee (DEFAULT_TABLE table=test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:53:39.167133 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb4d3f3f1b0416ebb7568c15e670cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:39.167439 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb4d3f3f1b0416ebb7568c15e670cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:39.173673 8589 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Bootstrap starting.
I20260619 00:53:39.173863 8591 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Bootstrap starting.
I20260619 00:53:39.174355 7983 catalog_manager.cc:4841] TS 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061): tablet 116d740b653341a8a4ccaad80e9c957a (table test-table [id=baba0498a4a74dbb871a0f22350988d6]) successfully deleted
I20260619 00:53:39.175455 8589 tablet_bootstrap.cc:654] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:39.175446 8591 tablet_bootstrap.cc:654] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:39.177909 8590 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Bootstrap starting.
I20260619 00:53:39.179632 8590 tablet_bootstrap.cc:654] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:39.183688 7983 catalog_manager.cc:4841] TS 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389): tablet 116d740b653341a8a4ccaad80e9c957a (table test-table [id=baba0498a4a74dbb871a0f22350988d6]) successfully deleted
I20260619 00:53:39.184252 8590 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: No bootstrap required, opened a new log
I20260619 00:53:39.184432 8590 ts_tablet_manager.cc:1397] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 00:53:39.185194 8590 raft_consensus.cc:348] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.185475 8590 raft_consensus.cc:374] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:39.185550 8590 raft_consensus.cc:729] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c4332fd3848a09b51175daef86eb6, State: Initialized, Role: FOLLOWER
I20260619 00:53:39.185586 8589 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: No bootstrap required, opened a new log
I20260619 00:53:39.185750 8591 tablet_bootstrap.cc:492] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: No bootstrap required, opened a new log
I20260619 00:53:39.185838 8590 consensus_queue.cc:260] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.185971 8591 ts_tablet_manager.cc:1397] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260619 00:53:39.186848 8591 raft_consensus.cc:348] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.187000 8589 ts_tablet_manager.cc:1397] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260619 00:53:39.187160 8591 raft_consensus.cc:374] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:39.187340 8591 raft_consensus.cc:729] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f80c908c21b4a74a93aaee6798b10d4, State: Initialized, Role: FOLLOWER
I20260619 00:53:39.187690 8591 consensus_queue.cc:260] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.187855 8589 raft_consensus.cc:348] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.188120 8589 raft_consensus.cc:374] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:39.188201 8589 raft_consensus.cc:729] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Initialized, Role: FOLLOWER
I20260619 00:53:39.188494 8589 consensus_queue.cc:260] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.189718 8589 ts_tablet_manager.cc:1428] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 00:53:39.188879 8590 ts_tablet_manager.cc:1428] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 00:53:39.190946 8591 ts_tablet_manager.cc:1428] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260619 00:53:39.531963 8596 raft_consensus.cc:482] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:53:39.532153 8596 raft_consensus.cc:504] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.532871 8596 leader_election.cc:288] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:39.533452 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb4d3f3f1b0416ebb7568c15e670cee" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4" is_pre_election: true
I20260619 00:53:39.533501 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb4d3f3f1b0416ebb7568c15e670cee" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6" is_pre_election: true
I20260619 00:53:39.533740 8297 raft_consensus.cc:2454] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:39.533814 8431 raft_consensus.cc:2454] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 0.
I20260619 00:53:39.534225 8083 leader_election.cc:302] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:39.534497 8596 raft_consensus.cc:2781] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:53:39.534610 8596 raft_consensus.cc:482] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:53:39.534691 8596 raft_consensus.cc:3037] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.536640 8596 raft_consensus.cc:504] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.537302 8596 leader_election.cc:288] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Requested vote from peers 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795), 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389)
I20260619 00:53:39.537819 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb4d3f3f1b0416ebb7568c15e670cee" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f80c908c21b4a74a93aaee6798b10d4"
I20260619 00:53:39.537822 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb4d3f3f1b0416ebb7568c15e670cee" candidate_uuid: "786a9a27af0d44148b89377ae3f89a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626c4332fd3848a09b51175daef86eb6"
I20260619 00:53:39.538039 8297 raft_consensus.cc:3037] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.538058 8431 raft_consensus.cc:3037] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:39.541129 8297 raft_consensus.cc:2454] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:39.541129 8431 raft_consensus.cc:2454] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786a9a27af0d44148b89377ae3f89a35 in term 1.
I20260619 00:53:39.541694 8086 leader_election.cc:302] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f80c908c21b4a74a93aaee6798b10d4, 626c4332fd3848a09b51175daef86eb6, 786a9a27af0d44148b89377ae3f89a35; no voters:
I20260619 00:53:39.542102 8596 raft_consensus.cc:2781] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:39.542280 8596 raft_consensus.cc:686] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Becoming Leader. State: Replica: 786a9a27af0d44148b89377ae3f89a35, State: Running, Role: LEADER
I20260619 00:53:39.542601 8596 consensus_queue.cc:237] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } }
I20260619 00:53:39.545293 7995 catalog_manager.cc:5495] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "786a9a27af0d44148b89377ae3f89a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786a9a27af0d44148b89377ae3f89a35" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20260619 00:53:39.647181 8297 raft_consensus.cc:1261] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:39.647192 8431 raft_consensus.cc:1261] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Refusing update from remote peer 786a9a27af0d44148b89377ae3f89a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:53:39.647760 8536 consensus_queue.cc:1035] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626c4332fd3848a09b51175daef86eb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:39.648017 8596 consensus_queue.cc:1035] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f80c908c21b4a74a93aaee6798b10d4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:53:39.695932 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:39.696135 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:53:39.699123 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8aac1a32fcc4a37a4842af1127ed956: Sending DeleteTablet for 3 replicas of tablet 6bb4d3f3f1b0416ebb7568c15e670cee
I20260619 00:53:39.700014 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260619 00:53:39.700469 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48240
I20260619 00:53:39.700619 8612 tablet_replica.cc:330] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: stopping tablet replica
I20260619 00:53:39.700839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8067
I20260619 00:53:39.700879 8612 raft_consensus.cc:2229] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:39.701129 8612 raft_consensus.cc:2258] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.701279 8613 tablet_replica.cc:330] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: stopping tablet replica
I20260619 00:53:39.701563 8613 raft_consensus.cc:2229] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:53:39.701941 8613 raft_consensus.cc:2258] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.703045 8613 ts_tablet_manager.cc:1905] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.708425 8612 ts_tablet_manager.cc:1905] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.709236 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb4d3f3f1b0416ebb7568c15e670cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:53:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260619 00:53:39.712709 8613 ts_tablet_manager.cc:1918] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:39.712920 8613 log.cc:1196] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/6bb4d3f3f1b0416ebb7568c15e670cee
I20260619 00:53:39.713590 8613 ts_tablet_manager.cc:1939] T 6bb4d3f3f1b0416ebb7568c15e670cee P 786a9a27af0d44148b89377ae3f89a35: Deleting consensus metadata
I20260619 00:53:39.714864 8616 tablet_replica.cc:330] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: stopping tablet replica
I20260619 00:53:39.715204 7983 catalog_manager.cc:4841] TS 786a9a27af0d44148b89377ae3f89a35 (127.0.97.1:40061): tablet 6bb4d3f3f1b0416ebb7568c15e670cee (table test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]) successfully deleted
I20260619 00:53:39.715339 8616 raft_consensus.cc:2229] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:53:39.715749 8616 raft_consensus.cc:2258] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:53:39.716786 8616 ts_tablet_manager.cc:1905] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:53:39.723217 8612 ts_tablet_manager.cc:1918] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:39.723383 8612 log.cc:1196] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/6bb4d3f3f1b0416ebb7568c15e670cee
I20260619 00:53:39.723766 8612 ts_tablet_manager.cc:1939] T 6bb4d3f3f1b0416ebb7568c15e670cee P 5f80c908c21b4a74a93aaee6798b10d4: Deleting consensus metadata
I20260619 00:53:39.724897 7980 catalog_manager.cc:4841] TS 5f80c908c21b4a74a93aaee6798b10d4 (127.0.97.2:38795): tablet 6bb4d3f3f1b0416ebb7568c15e670cee (table test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]) successfully deleted
I20260619 00:53:39.725258 8616 ts_tablet_manager.cc:1918] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:53:39.725390 8616 log.cc:1196] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/6bb4d3f3f1b0416ebb7568c15e670cee
I20260619 00:53:39.725931 8616 ts_tablet_manager.cc:1939] T 6bb4d3f3f1b0416ebb7568c15e670cee P 626c4332fd3848a09b51175daef86eb6: Deleting consensus metadata
I20260619 00:53:39.727070 7983 catalog_manager.cc:4841] TS 626c4332fd3848a09b51175daef86eb6 (127.0.97.3:44389): tablet 6bb4d3f3f1b0416ebb7568c15e670cee (table test-table [id=d5ff858a0e3245bfa869a1c8d51214d7]) successfully deleted
Jun 19 00:53:39 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 00:53:39.744381 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.701782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.0.97.1:40061)
0619 00:53:39.714970 (+ 13188us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:39.714977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:39.719622 (+ 4645us) server_negotiation.cc:407] Connection header received
0619 00:53:39.719835 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:39.719841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:39.719928 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:39.720022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:39.726669 (+ 6647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.727689 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.729177 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.729951 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.730075 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.735792 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:39.735816 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:39.735837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:39.735911 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:39.740630 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:39.741572 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:39.741584 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:39.741589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:39.741767 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:39.742245 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:39.742250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:39.742254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:39.742744 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:39.743083 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:39.743705 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 00:53:39.743950 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12724,"thread_start_us":12198,"threads_started":1}
I20260619 00:53:39.745252 8198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:39.888199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8067
I20260619 00:53:39.912302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8207
Jun 19 00:53:39 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 00:53:39.928208 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:39.913234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.97.2:38795)
0619 00:53:39.913758 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:39.913763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:39.913778 (+ 15us) server_negotiation.cc:407] Connection header received
0619 00:53:39.913830 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:39.913834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:39.913897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:39.913991 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:39.915356 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.916140 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.917443 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:39.917866 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:39.917970 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:39.922066 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:39.922082 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:39.922097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:39.922146 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:39.925491 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:39.926195 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:39.926202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:39.926205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:39.926380 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:39.926762 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:39.926766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:39.926768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:39.927119 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:39.927327 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:39.927656 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 00:53:39.927828 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":267,"threads_started":1}
I20260619 00:53:39.928902 8337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:40.026237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8207
I20260619 00:53:40.045370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8346
Jun 19 00:53:40 dist-test-slave-r7bh krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 00:53:40.063210 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:40.046077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.0.97.3:44389)
0619 00:53:40.046534 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:40.046538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:40.046555 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:40.046607 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:40.046611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:40.046673 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:40.046763 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:40.048061 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:40.048917 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:40.050670 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:40.051171 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:40.051267 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:40.056715 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:40.056732 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:40.056749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:40.056788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:40.060086 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:40.060848 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:40.060859 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:40.060863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:40.061064 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:40.061481 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:40.061488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:40.061492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:40.061920 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:40.062224 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:40.062557 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 00:53:40.062815 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260619 00:53:40.063884 8475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:40.208844 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8346
I20260619 00:53:40.234441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 7964
I20260619 00:53:40.235738 8036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:40.352504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 7964
2026-06-19T00:53:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6058 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:40 dist-test-slave-r7bh krb5kdc[8627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:40 dist-test-slave-r7bh krb5kdc[8627](info): set up 2 sockets
Jun 19 00:53:40 dist-test-slave-r7bh krb5kdc[8627](info): commencing operation
krb5kdc: starting...
W20260619 00:53:41.485864 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:41 dist-test-slave-r7bh krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:41.711867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:42.044096 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:42.044472 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:42.044639 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:42.052714 8640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:42.052914 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:42.053031 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:42.053118 8640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:42.053191 8640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:42.063158 8640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:42.064563 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:42.065971 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:42.078027 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:42.078104 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:42.078052 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:42.078868 8640 server_base.cc:1034] running on GCE node
I20260619 00:53:42.079666 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:42.081146 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:42.082383 8640 hybrid_clock.cc:648] HybridClock initialized: now 1781830422082335 us; error 42 us; skew 500 ppm
I20260619 00:53:42.082733 8640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T00:53:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1796 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:53:42 dist-test-slave-r7bh krb5kdc[8654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:53:42 dist-test-slave-r7bh krb5kdc[8654](info): set up 2 sockets
Jun 19 00:53:42 dist-test-slave-r7bh krb5kdc[8654](info): commencing operation
krb5kdc: starting...
W20260619 00:53:43.261812 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:53:43 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:53:43.485927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45087
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:43.811435 8667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:43.811753 8667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:43.811843 8667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:43.819332 8667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:43.819453 8667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:43.819496 8667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:43.819552 8667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:43.819588 8667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:43.829416 8667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:43.830708 8667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:43.831972 8667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:43.844030 8672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:43.843983 8673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:43.843914 8675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:43.844836 8667 server_base.cc:1034] running on GCE node
I20260619 00:53:43.845650 8667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:43.847118 8667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:43.848357 8667 hybrid_clock.cc:648] HybridClock initialized: now 1781830423848311 us; error 50 us; skew 500 ppm
I20260619 00:53:43.848728 8667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:43 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:43.851831 8667 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:43.853463 8667 webserver.cc:460] Webserver started at http://127.0.97.62:40339/ using document root <none> and password file <none>
I20260619 00:53:43.854162 8667 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:43.854311 8667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:43.854630 8667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:43.857628 8667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "9d77e8b8dbcb4539abddfe49932abcb8"
format_stamp: "Formatted at 2026-06-19 00:53:43 on dist-test-slave-r7bh"
server_key: "f77f5098cd10a8d8ea96d3f8d00e7787"
server_key_iv: "2ec93d39a7b9720f2f5db124dcf51a39"
server_key_version: "encryptionkey@0"
I20260619 00:53:43.858494 8667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "9d77e8b8dbcb4539abddfe49932abcb8"
format_stamp: "Formatted at 2026-06-19 00:53:43 on dist-test-slave-r7bh"
server_key: "f77f5098cd10a8d8ea96d3f8d00e7787"
server_key_iv: "2ec93d39a7b9720f2f5db124dcf51a39"
server_key_version: "encryptionkey@0"
I20260619 00:53:43.863833 8667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:53:43.867488 8682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:43.869009 8667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:43.869202 8667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "9d77e8b8dbcb4539abddfe49932abcb8"
format_stamp: "Formatted at 2026-06-19 00:53:43 on dist-test-slave-r7bh"
server_key: "f77f5098cd10a8d8ea96d3f8d00e7787"
server_key_iv: "2ec93d39a7b9720f2f5db124dcf51a39"
server_key_version: "encryptionkey@0"
I20260619 00:53:43.869429 8667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:43.886386 8667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:43.891065 8667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:43.891374 8667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:43.909843 8667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45087
I20260619 00:53:43.909873 8733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45087 every 8 connection(s)
I20260619 00:53:43.911337 8667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:43.915200 8734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:53:43.917300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8667
I20260619 00:53:43.917517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:53:43.918298 388 external_mini_cluster.cc:1428] Setting key dd557ab2e73a82f2c0bcf9d2fa245dad
I20260619 00:53:43.928188 8734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Bootstrap starting.
I20260619 00:53:43.931720 8734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Neither blocks nor log segments found. Creating new log.
I20260619 00:53:43.932669 8734 log.cc:824] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:43.935837 8734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: No bootstrap required, opened a new log
Jun 19 00:53:43 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:43.944171 8734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:43.944622 8734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:53:43.944777 8734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d77e8b8dbcb4539abddfe49932abcb8, State: Initialized, Role: FOLLOWER
I20260619 00:53:43.945562 8734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:43.945858 8734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:43.946002 8734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:43.946214 8734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:53:43.948115 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:43.920468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.0.97.62:45087)
0619 00:53:43.921309 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:43.921324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:43.921370 (+ 46us) server_negotiation.cc:407] Connection header received
0619 00:53:43.923033 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:43.923054 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:43.923459 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:43.923810 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:43.925665 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:43.927062 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:43.928925 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:43.929609 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:43.929765 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:43.939676 (+ 9911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:43.939685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:43.939703 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:43.939762 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:43.943444 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:43.944265 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:43.944272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:43.944280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:43.944472 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:43.944987 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:43.944997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:43.945000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:43.945595 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:43.945948 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:43.946440 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 00:53:43.946957 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":399,"threads_started":1}
I20260619 00:53:43.948940 8734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:43.949535 8734 leader_election.cc:302] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d77e8b8dbcb4539abddfe49932abcb8; no voters:
I20260619 00:53:43.950227 8734 leader_election.cc:288] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:53:43.950403 8739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:53:43.951105 8739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 LEADER]: Becoming Leader. State: Replica: 9d77e8b8dbcb4539abddfe49932abcb8, State: Running, Role: LEADER
I20260619 00:53:43.952353 8739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:43.953191 8734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:43.955456 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d77e8b8dbcb4539abddfe49932abcb8. Latest consensus state: current_term: 1 leader_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } } }
I20260619 00:53:43.957021 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:43.955997 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } } }
I20260619 00:53:43.957732 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:43.958515 8746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:43.965927 8746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:43.978953 8746 catalog_manager.cc:1347] Generated new cluster ID: d4e81e2a878249fcbfdae111026d7725
I20260619 00:53:43.979105 8746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:43.996989 8746 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:53:43.998965 8746 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:44.007321 8746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Generated new TSK 0
I20260619 00:53:44.008471 8746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:53:44.069115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:44.406695 8760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:44.407006 8760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:44.407181 8760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:44.414655 8760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:44.414788 8760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:44.414994 8760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:44.424942 8760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:44.426277 8760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:44.427621 8760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:44.445370 8768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:44.445379 8766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:44.445286 8765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:44.446123 8760 server_base.cc:1034] running on GCE node
I20260619 00:53:44.446750 8760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:44.447965 8760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:44.449177 8760 hybrid_clock.cc:648] HybridClock initialized: now 1781830424449133 us; error 37 us; skew 500 ppm
I20260619 00:53:44.449558 8760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:44 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:44.452878 8760 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:53:44.454366 8760 webserver.cc:460] Webserver started at http://127.0.97.1:46131/ using document root <none> and password file <none>
I20260619 00:53:44.455086 8760 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:44.455214 8760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:44.455546 8760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:44.458813 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee085c593f7842378da2b2e7157d2462"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "6280610d30e61a0fa4e7718f7eeaaadd"
server_key_iv: "d6538f347b2cef81bca5b85fab2095b8"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.459645 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee085c593f7842378da2b2e7157d2462"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "6280610d30e61a0fa4e7718f7eeaaadd"
server_key_iv: "d6538f347b2cef81bca5b85fab2095b8"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.464956 8760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:53:44.468603 8775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:44.470153 8760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:44.470386 8760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "ee085c593f7842378da2b2e7157d2462"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "6280610d30e61a0fa4e7718f7eeaaadd"
server_key_iv: "d6538f347b2cef81bca5b85fab2095b8"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.470607 8760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:44.488478 8760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:44.493151 8760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:44.493459 8760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:44.494472 8760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:44.495882 8760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:44.495981 8760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:44.496110 8760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:44.496191 8760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:44.530150 8760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36081
I20260619 00:53:44.530191 8887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36081 every 8 connection(s)
I20260619 00:53:44.531481 8760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:44.531919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8760
I20260619 00:53:44.532117 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:53:44.532994 388 external_mini_cluster.cc:1428] Setting key 48aa4b271acc30258ecd5ba554c080f7
Jun 19 00:53:44 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:44.552892 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:44.534276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38773 (local address 127.0.97.62:45087)
0619 00:53:44.534748 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:44.534754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:44.536004 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 00:53:44.537961 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:44.537966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:44.538053 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:44.538191 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:44.539795 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:44.540525 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:44.542267 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:44.542697 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:44.542848 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:44.546772 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:44.546778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:44.546781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:44.546818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:44.549617 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:44.550482 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:44.550487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:44.550489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:44.550611 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:44.551090 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:44.551095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:44.551097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:44.551378 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:44.551532 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:44.552274 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 00:53:44.552506 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:53:44.554068 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:44.534999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:38773)
0619 00:53:44.535836 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:44.535883 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:44.537658 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:44.538450 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:44.538459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:44.539019 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:44.539615 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:44.539632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:44.540704 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:44.540708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:44.542108 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:44.542121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:44.542975 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:44.542984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:44.543231 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:44.543990 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:44.544018 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:44.546498 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:44.549824 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:44.549835 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:44.549859 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:44.550325 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:44.550734 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:44.550740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:44.550743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:44.550957 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:44.551514 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:44.551522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:44.551897 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 00:53:44.552173 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 00:53:44.552738 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":348,"threads_started":1}
I20260619 00:53:44.555902 8888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45087
I20260619 00:53:44.556180 8888 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:53:44.557087 8888 heartbeater.cc:511] Master 127.0.97.62:45087 requested a full tablet report, sending...
I20260619 00:53:44.559310 8699 ts_manager.cc:194] Registered new tserver with Master: ee085c593f7842378da2b2e7157d2462 (127.0.97.1:36081)
I20260619 00:53:44.560801 8699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:53:44.585232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:44.915427 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:44.915867 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:44.916039 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:44.923484 8894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:44.923660 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:44.923833 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:44.933831 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:44.935204 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:44.936542 8894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:44.949324 8899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:44.949416 8900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:44.949352 8902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:44.950441 8894 server_base.cc:1034] running on GCE node
I20260619 00:53:44.951087 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:44.952052 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:44.953292 8894 hybrid_clock.cc:648] HybridClock initialized: now 1781830424953229 us; error 59 us; skew 500 ppm
I20260619 00:53:44.953680 8894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:44 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:44.957000 8894 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:53:44.958528 8894 webserver.cc:460] Webserver started at http://127.0.97.2:40745/ using document root <none> and password file <none>
I20260619 00:53:44.959231 8894 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:44.959342 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:44.959615 8894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:44.962791 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "ba02556738a14e238e66850b6fb2a9d9"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "98a31d912d0a1af191e610e4cb057fb4"
server_key_iv: "7f45e73c29ad600082169726c071e0a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.963640 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ba02556738a14e238e66850b6fb2a9d9"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "98a31d912d0a1af191e610e4cb057fb4"
server_key_iv: "7f45e73c29ad600082169726c071e0a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.969683 8894 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 00:53:44.973615 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:44.975356 8894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:53:44.975503 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "ba02556738a14e238e66850b6fb2a9d9"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "98a31d912d0a1af191e610e4cb057fb4"
server_key_iv: "7f45e73c29ad600082169726c071e0a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:44.975685 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:44.997234 8894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:45.002244 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:45.002522 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:45.003585 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:45.005035 8894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:45.005119 8894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:45.005230 8894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:45.005280 8894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:45.041329 8894 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36559
I20260619 00:53:45.041363 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36559 every 8 connection(s)
I20260619 00:53:45.042757 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:45.048264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 8894
I20260619 00:53:45.048441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:53:45.048935 388 external_mini_cluster.cc:1428] Setting key b28937bb072030dbbbcc3acee12f559e
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:45.063850 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.045503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58325 (local address 127.0.97.62:45087)
0619 00:53:45.045685 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:45.045690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:45.046935 (+ 1245us) server_negotiation.cc:407] Connection header received
0619 00:53:45.048923 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:45.048929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:45.049051 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:45.049177 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:45.051214 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.051918 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.053627 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.054125 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.054256 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.058018 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:45.058024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:45.058027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:45.058059 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:45.060667 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.061471 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.061478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.061486 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.061608 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.062065 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.062075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.062077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.062340 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:45.062494 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:45.063255 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 00:53:45.063496 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:53:45.065062 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.046038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:58325)
0619 00:53:45.046758 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:45.046804 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:45.048604 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:45.049451 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:45.049466 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:45.050212 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:45.051046 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:45.051074 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.052059 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.052064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:45.053491 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:45.053502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.054348 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.054355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:45.054636 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.055387 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:45.055413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:45.057775 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:45.060847 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:45.060857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:45.060877 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:45.061333 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:45.061726 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:45.061731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:45.061735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:45.061939 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:45.062469 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:45.062478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:45.062852 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 00:53:45.063155 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 00:53:45.063739 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":278,"threads_started":1}
I20260619 00:53:45.066921 9022 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45087
I20260619 00:53:45.067250 9022 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:45.068019 9022 heartbeater.cc:511] Master 127.0.97.62:45087 requested a full tablet report, sending...
I20260619 00:53:45.069741 8699 ts_manager.cc:194] Registered new tserver with Master: ba02556738a14e238e66850b6fb2a9d9 (127.0.97.2:36559)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:53:45.070665 8699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:53:45.098287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:45.424715 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:45.425028 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:45.425203 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:45.432512 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:45.432631 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:45.432830 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:45.442934 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:45.444242 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:45.445623 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:45.458977 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:45.459017 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:45.458979 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:45.459986 9027 server_base.cc:1034] running on GCE node
I20260619 00:53:45.460704 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:45.461755 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:45.462966 9027 hybrid_clock.cc:648] HybridClock initialized: now 1781830425462925 us; error 40 us; skew 500 ppm
I20260619 00:53:45.463305 9027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:45.466781 9027 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:53:45.468194 9027 webserver.cc:460] Webserver started at http://127.0.97.3:45587/ using document root <none> and password file <none>
I20260619 00:53:45.468926 9027 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:45.469031 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:45.469291 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:53:45.472108 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "596f342ca71c4a29a0b6b383953fd265"
format_stamp: "Formatted at 2026-06-19 00:53:45 on dist-test-slave-r7bh"
server_key: "7e7f8d4117d3310080d6f2f055cd48a1"
server_key_iv: "2be001b306ea614708f753a551c6e362"
server_key_version: "encryptionkey@0"
I20260619 00:53:45.473145 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "596f342ca71c4a29a0b6b383953fd265"
format_stamp: "Formatted at 2026-06-19 00:53:45 on dist-test-slave-r7bh"
server_key: "7e7f8d4117d3310080d6f2f055cd48a1"
server_key_iv: "2be001b306ea614708f753a551c6e362"
server_key_version: "encryptionkey@0"
I20260619 00:53:45.478619 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 00:53:45.482544 9042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:45.484081 9027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:53:45.484333 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "596f342ca71c4a29a0b6b383953fd265"
format_stamp: "Formatted at 2026-06-19 00:53:45 on dist-test-slave-r7bh"
server_key: "7e7f8d4117d3310080d6f2f055cd48a1"
server_key_iv: "2be001b306ea614708f753a551c6e362"
server_key_version: "encryptionkey@0"
I20260619 00:53:45.484561 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:45.501528 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:45.506186 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:45.506503 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:45.507570 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:45.509079 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:45.509186 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:45.509382 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:45.509466 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:45.542526 9027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40403
I20260619 00:53:45.542608 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40403 every 8 connection(s)
I20260619 00:53:45.543816 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:45.550573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9027
I20260619 00:53:45.550766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:53:45.551163 388 external_mini_cluster.cc:1428] Setting key 5455a76b3df91b2aaafcd8da7fe7628b
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:53:45.564064 8888 heartbeater.cc:503] Master 127.0.97.62:45087 was elected leader, sending a full tablet report...
I20260619 00:53:45.564922 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.546563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57215 (local address 127.0.97.62:45087)
0619 00:53:45.546791 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:45.546797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:45.548126 (+ 1329us) server_negotiation.cc:407] Connection header received
0619 00:53:45.550037 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:45.550044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:45.550180 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:45.550311 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:45.552198 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.553248 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.555110 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.555553 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.555702 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.559330 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:45.559335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:45.559339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:45.559372 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:45.561794 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.562545 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.562558 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.562561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.562686 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.563147 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.563154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.563157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.563427 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:45.563576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:45.564296 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 00:53:45.564521 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:53:45.566357 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.547203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:57215)
0619 00:53:45.547932 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:45.547991 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:45.549694 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:45.550586 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:45.550596 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:45.551357 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:45.551986 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:45.552005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.553418 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.553423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:45.554947 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:45.554960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.555821 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.555828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:45.556082 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.556757 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:45.556780 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:45.559039 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:53:45.561945 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:45.561954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:45.561973 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:45.562414 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:45.562818 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:53:45.562824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:53:45.562828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:53:45.563036 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:53:45.563552 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:53:45.563560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:53:45.563924 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 00:53:45.564201 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 00:53:45.564751 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
I20260619 00:53:45.567857 9155 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45087
I20260619 00:53:45.568185 9155 heartbeater.cc:464] Registering TS with master...
I20260619 00:53:45.569038 9155 heartbeater.cc:511] Master 127.0.97.62:45087 requested a full tablet report, sending...
I20260619 00:53:45.570524 8699 ts_manager.cc:194] Registered new tserver with Master: 596f342ca71c4a29a0b6b383953fd265 (127.0.97.3:40403)
I20260619 00:53:45.571329 8699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57215
I20260619 00:53:45.579437 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 00:53:45.719241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8760
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:53:45.734923 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.720309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.97.1:36081)
0619 00:53:45.720963 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:45.720970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:45.720999 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:53:45.721067 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:45.721073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:45.721289 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:45.721431 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:45.722714 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.723723 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.724991 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.725567 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.725665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.729667 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:45.729680 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:45.729694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:45.729737 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:45.732609 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.733244 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.733251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.733253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.733385 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.733679 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.733681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.733683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.734021 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:45.734225 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:45.734445 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 00:53:45.734641 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":296,"threads_started":1}
I20260619 00:53:45.736178 8883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:45.818511 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8760
I20260619 00:53:45.834416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8894
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:53:45.850641 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.835460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.97.2:36559)
0619 00:53:45.836334 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:45.836343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:45.836377 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:53:45.836443 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:45.836449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:45.836653 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:45.836860 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:45.838045 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.839058 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.840287 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.840921 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.841050 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.845261 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:45.845272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:45.845286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:45.845324 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:45.848010 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.848628 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.848635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.848637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.848784 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.849247 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.849251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.849253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.849574 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:45.849799 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:45.850031 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 00:53:45.850251 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":613,"threads_started":1}
I20260619 00:53:45.851927 9017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:45.934499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8894
I20260619 00:53:45.950202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9027
Jun 19 00:53:45 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:53:45.967545 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:45.951244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.0.97.3:40403)
0619 00:53:45.951848 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:45.951855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:45.951884 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:53:45.951943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:45.951948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:45.952154 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:45.952340 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:45.953854 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.955024 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.956325 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:45.957055 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:45.957195 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:45.961603 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:45.961616 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:45.961633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:45.961671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:45.964488 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.965357 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.965368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.965372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.965514 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:45.965921 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:45.965925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:45.965927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:45.966270 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:45.966592 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:45.966930 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 00:53:45.967197 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":298,"threads_started":1}
I20260619 00:53:45.968971 9150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:46.051966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9027
I20260619 00:53:46.068297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 8667
I20260619 00:53:46.069641 8729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:53:46.170759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 8667
I20260619 00:53:46.191244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45087
--webserver_interface=127.0.97.62
--webserver_port=40339
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:53:46.521495 9172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:46.521806 9172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:46.522019 9172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:46.529506 9172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:53:46.529629 9172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:46.529709 9172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:46.529814 9172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:53:46.529887 9172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:53:46.539966 9172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:46.541438 9172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:46.542851 9172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:46.555059 9178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:46.555121 9180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:46.555142 9177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:46.556252 9172 server_base.cc:1034] running on GCE node
I20260619 00:53:46.557183 9172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:46.558673 9172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:46.559895 9172 hybrid_clock.cc:648] HybridClock initialized: now 1781830426559830 us; error 66 us; skew 500 ppm
I20260619 00:53:46.560231 9172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:46 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:46.563401 9172 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:53:46.564945 9172 webserver.cc:460] Webserver started at http://127.0.97.62:40339/ using document root <none> and password file <none>
I20260619 00:53:46.565588 9172 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:46.565699 9172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:46.571034 9172 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 00:53:46.573968 9187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:46.575487 9172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:53:46.575695 9172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "9d77e8b8dbcb4539abddfe49932abcb8"
format_stamp: "Formatted at 2026-06-19 00:53:43 on dist-test-slave-r7bh"
server_key: "f77f5098cd10a8d8ea96d3f8d00e7787"
server_key_iv: "2ec93d39a7b9720f2f5db124dcf51a39"
server_key_version: "encryptionkey@0"
I20260619 00:53:46.576659 9172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:46.588482 9172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:46.592538 9172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:46.592895 9172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:46.611575 9172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45087
I20260619 00:53:46.611617 9238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45087 every 8 connection(s)
I20260619 00:53:46.613162 9172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:53:46.619952 9239 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:53:46.623340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9172
I20260619 00:53:46.623502 9239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Bootstrap starting.
I20260619 00:53:46.624265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36081
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45087
--webserver_port=46131
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:53:46.644929 9239 log.cc:824] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:53:46.654187 9239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:53:46.654736 9239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Bootstrap complete.
I20260619 00:53:46.663841 9239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:46.664666 9239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d77e8b8dbcb4539abddfe49932abcb8, State: Initialized, Role: FOLLOWER
I20260619 00:53:46.665347 9239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:46.665583 9239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:53:46.665700 9239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:53:46.665915 9239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:53:46.669642 9239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:46.670145 9239 leader_election.cc:302] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d77e8b8dbcb4539abddfe49932abcb8; no voters:
I20260619 00:53:46.670854 9239 leader_election.cc:288] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 00:53:46.671051 9243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 00:53:46.672322 9243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [term 2 LEADER]: Becoming Leader. State: Replica: 9d77e8b8dbcb4539abddfe49932abcb8, State: Running, Role: LEADER
I20260619 00:53:46.672982 9243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } }
I20260619 00:53:46.673234 9239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:53:46.676203 9245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d77e8b8dbcb4539abddfe49932abcb8. Latest consensus state: current_term: 2 leader_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } } }
I20260619 00:53:46.676295 9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d77e8b8dbcb4539abddfe49932abcb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45087 } } }
I20260619 00:53:46.676699 9245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:46.676709 9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:53:46.678023 9251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:53:46.692673 9251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:53:46.694202 9251 catalog_manager.cc:1259] Loaded cluster ID: d4e81e2a878249fcbfdae111026d7725
I20260619 00:53:46.694376 9251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:53:46.698170 9251 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:53:46.699682 9251 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d77e8b8dbcb4539abddfe49932abcb8: Loaded TSK: 0
I20260619 00:53:46.701452 9251 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 00:53:46.984999 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:46.985360 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:46.985532 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:46.993611 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:46.993783 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:46.993965 9241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:53:47.005033 9241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46131
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:47.006475 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:47.007918 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:47.021786 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.022347 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.022434 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:47.023147 9241 server_base.cc:1034] running on GCE node
I20260619 00:53:47.023938 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:47.025074 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:47.026329 9241 hybrid_clock.cc:648] HybridClock initialized: now 1781830427026276 us; error 52 us; skew 500 ppm
I20260619 00:53:47.026722 9241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830427, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:47.030508 9241 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 00:53:47.032251 9241 webserver.cc:460] Webserver started at http://127.0.97.1:46131/ using document root <none> and password file <none>
I20260619 00:53:47.033133 9241 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:47.033315 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:47.039597 9241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 00:53:47.043121 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.044750 9241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:47.045042 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "ee085c593f7842378da2b2e7157d2462"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "6280610d30e61a0fa4e7718f7eeaaadd"
server_key_iv: "d6538f347b2cef81bca5b85fab2095b8"
server_key_version: "encryptionkey@0"
I20260619 00:53:47.046060 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:47.063603 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:47.068194 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:47.068504 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:47.069657 9241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:47.071069 9241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:47.071205 9241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.071348 9241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:47.071436 9241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.106024 9241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36081
I20260619 00:53:47.106082 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36081 every 8 connection(s)
I20260619 00:53:47.107556 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:53:47.110107 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9241
I20260619 00:53:47.110911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36559
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45087
--webserver_port=40745
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:47.130194 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.112225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:35393)
0619 00:53:47.113454 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.113505 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.116181 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.117888 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.117902 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.118520 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.119076 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.119094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.121619 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.121623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.122891 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.122914 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.124453 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.124458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.124670 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.125454 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.125477 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.129127 (+ 3650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":725,"threads_started":1}
W20260619 00:53:47.131138 9388 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45087 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45087: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
W20260619 00:53:47.131608 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.112186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35393 (local address 127.0.97.62:45087)
0619 00:53:47.113128 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.113145 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.113680 (+ 535us) server_negotiation.cc:407] Connection header received
0619 00:53:47.116922 (+ 3242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.116944 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.117339 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.117655 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.119290 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.121428 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.123064 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.124226 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.124430 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.129656 (+ 5226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35393: BlockingRecv error: recv got EOF from 127.0.97.1:35393 (error 108)
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":309,"threads_started":1}
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:47.141052 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.131818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:55543)
0619 00:53:47.132220 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.132244 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.132378 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.132692 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.132700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.133193 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.133430 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.133447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.134467 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.134475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.136147 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.136161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.136806 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.136833 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.137010 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.137625 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.137647 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.140710 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259}
W20260619 00:53:47.141165 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.132246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55543 (local address 127.0.97.62:45087)
0619 00:53:47.132417 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.132421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.132437 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:53:47.132485 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.132489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.132543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.132626 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.133579 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.134313 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.136284 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.136674 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.136764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.140943 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55543: BlockingRecv error: recv got EOF from 127.0.97.1:55543 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:47.150487 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.141892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:52435)
0619 00:53:47.142081 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.142101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.142208 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.142563 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.142566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.142830 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.142984 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.142991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.144414 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.144418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.145383 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.145395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.146368 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.146372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.146583 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.147218 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.147235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.150191 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 00:53:47.150635 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.142081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52435 (local address 127.0.97.62:45087)
0619 00:53:47.142263 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.142269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.142287 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:53:47.142346 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.142351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.142437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.142535 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.143119 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.144242 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.145520 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.146207 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.146343 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.150397 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52435: BlockingRecv error: recv got EOF from 127.0.97.1:52435 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 00:53:47.150805 9388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 00:53:47.445878 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:47.446167 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:47.446262 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:47.453554 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:47.453675 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:47.453890 9392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:53:47.463807 9392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40745
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:47.465226 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:47.466500 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:47.479883 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.479913 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.479949 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:47.480717 9392 server_base.cc:1034] running on GCE node
I20260619 00:53:47.481372 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:47.482407 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:47.483652 9392 hybrid_clock.cc:648] HybridClock initialized: now 1781830427483598 us; error 46 us; skew 500 ppm
I20260619 00:53:47.484030 9392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830427, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:47.487464 9392 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 00:53:47.488991 9392 webserver.cc:460] Webserver started at http://127.0.97.2:40745/ using document root <none> and password file <none>
I20260619 00:53:47.489851 9392 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:47.489969 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:47.495618 9392 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:53:47.498518 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.500015 9392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:53:47.500157 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "ba02556738a14e238e66850b6fb2a9d9"
format_stamp: "Formatted at 2026-06-19 00:53:44 on dist-test-slave-r7bh"
server_key: "98a31d912d0a1af191e610e4cb057fb4"
server_key_iv: "7f45e73c29ad600082169726c071e0a2"
server_key_version: "encryptionkey@0"
I20260619 00:53:47.501194 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:47.532050 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:47.537092 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:47.537395 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:47.538358 9392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:47.539662 9392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:47.539762 9392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.539834 9392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:47.539883 9392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.572674 9392 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36559
I20260619 00:53:47.572765 9519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36559 every 8 connection(s)
I20260619 00:53:47.574344 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:53:47.576318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9392
I20260619 00:53:47.577008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40403
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45087
--webserver_port=45587
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 00:53:47.592860 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.577069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60383 (local address 127.0.97.62:45087)
0619 00:53:47.577283 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.577291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.578717 (+ 1426us) server_negotiation.cc:407] Connection header received
0619 00:53:47.581745 (+ 3028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.581753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.581846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.581943 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.584195 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.585014 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.586765 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.587164 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.587301 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.592564 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60383: BlockingRecv error: recv got EOF from 127.0.97.2:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 00:53:47.593628 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.577691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:60383)
0619 00:53:47.578545 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.578598 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.581349 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.582243 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.582257 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.583135 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.583979 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.584007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.585202 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.585211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.586629 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.586641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.587415 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.587425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.587676 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.588404 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.588428 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.592364 (+ 3936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
W20260619 00:53:47.594398 9520 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45087 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45087: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:47.602955 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.595055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:42781)
0619 00:53:47.595233 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.595254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.595384 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.595727 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.595732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.596063 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.596201 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.596209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.597261 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.597267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.598142 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.598153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.598867 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.598873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.599043 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.599654 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.599671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.602645 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 00:53:47.603111 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.595160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42781 (local address 127.0.97.62:45087)
0619 00:53:47.595337 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.595342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.595369 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:53:47.595532 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.595537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.595606 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.595705 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.596342 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.597131 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.598289 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.598707 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.598833 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.602867 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42781: BlockingRecv error: recv got EOF from 127.0.97.2:42781 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:47.613454 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:47.603753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:35207)
0619 00:53:47.603956 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:47.603974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:47.604079 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:47.604555 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:47.604560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:47.604986 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:47.605178 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.605189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.606364 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.606370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.607904 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:47.607919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.608668 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.608675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:47.608963 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.609576 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:47.609599 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:47.613161 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 00:53:47.613634 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:47.603907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35207 (local address 127.0.97.62:45087)
0619 00:53:47.604055 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:47.604059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:47.604076 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:53:47.604268 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:47.604272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:47.604358 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:47.604530 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:47.605348 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.606227 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.608061 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:47.608508 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:47.608622 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:47.613416 (+ 4794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35207: BlockingRecv error: recv got EOF from 127.0.97.2:35207 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 00:53:47.614085 9520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 00:53:47.921391 9523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:53:47.921694 9523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:53:47.921854 9523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:53:47.929369 9523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:53:47.929497 9523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:53:47.929611 9523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:53:47.939353 9523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45587
--tserver_master_addrs=127.0.97.62:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:53:47.940630 9523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:53:47.941977 9523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:53:47.955345 9528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.955647 9529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:53:47.955698 9531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:53:47.956736 9523 server_base.cc:1034] running on GCE node
I20260619 00:53:47.957471 9523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:53:47.958549 9523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:53:47.959822 9523 hybrid_clock.cc:648] HybridClock initialized: now 1781830427959777 us; error 73 us; skew 500 ppm
I20260619 00:53:47.960177 9523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:53:47 dist-test-slave-r7bh krb5kdc[8654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830427, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:53:47.963590 9523 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 00:53:47.965148 9523 webserver.cc:460] Webserver started at http://127.0.97.3:45587/ using document root <none> and password file <none>
I20260619 00:53:47.965857 9523 fs_manager.cc:359] Metadata directory not provided
I20260619 00:53:47.965974 9523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:53:47.971822 9523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:53:47.975086 9538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:47.976756 9523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:53:47.977032 9523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "596f342ca71c4a29a0b6b383953fd265"
format_stamp: "Formatted at 2026-06-19 00:53:45 on dist-test-slave-r7bh"
server_key: "7e7f8d4117d3310080d6f2f055cd48a1"
server_key_iv: "2be001b306ea614708f753a551c6e362"
server_key_version: "encryptionkey@0"
I20260619 00:53:47.978018 9523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:53:47.999486 9523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:53:48.004711 9523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:53:48.005070 9523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:53:48.006250 9523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:53:48.007692 9523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:53:48.007793 9523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:48.007942 9523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:53:48.008019 9523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:53:48.041742 9523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40403
I20260619 00:53:48.041795 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40403 every 8 connection(s)
I20260619 00:53:48.043280 9523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:53:48.044623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9523
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 00:53:48.060779 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:48.046871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41179 (local address 127.0.97.62:45087)
0619 00:53:48.047360 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.047368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.047957 (+ 589us) server_negotiation.cc:407] Connection header received
0619 00:53:48.049823 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.049829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.049905 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.050017 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.051841 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.053026 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.054748 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.055391 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.055557 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.060477 (+ 4920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41179: BlockingRecv error: recv got EOF from 127.0.97.3:41179 (error 108)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260619 00:53:48.061192 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.045706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.97.62:45087)
0619 00:53:48.045941 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.045950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.046428 (+ 478us) server_negotiation.cc:407] Connection header received
0619 00:53:48.046600 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.046604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.046668 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.046789 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.048476 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.049309 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.050913 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.051322 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.051470 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.054400 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:53:48.054408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:53:48.054424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:53:48.054458 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:53:48.057599 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:48.058424 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:48.058433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:48.058443 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:48.058635 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:53:48.059038 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:53:48.059043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:53:48.059047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:53:48.059503 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:53:48.059859 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 00:53:48.060310 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 00:53:48.060708 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:53:48.061601 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.046917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:41179)
0619 00:53:48.047790 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:48.047835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:48.049595 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:48.050355 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:48.050364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:48.050982 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:48.051659 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.051677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.053194 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.053198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.054564 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.054577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.055669 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.055678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.055925 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.056585 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:48.056610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:48.060238 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":349,"threads_started":1}
W20260619 00:53:48.062269 9651 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45087 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45087: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:48.071062 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.063000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:43811)
0619 00:53:48.063159 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:48.063185 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:48.063273 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:48.063625 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:48.063629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:48.064066 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:48.064206 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.064214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.065507 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.065511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.066415 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.066427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.067057 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.067061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.067185 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.067738 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:48.067755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:48.070839 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:53:48.071345 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:48.063059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43811 (local address 127.0.97.62:45087)
0619 00:53:48.063217 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.063222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.063241 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:53:48.063381 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.063385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.063473 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.063576 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.064344 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.065389 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.066533 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.066929 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.067019 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.071053 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43811: BlockingRecv error: recv got EOF from 127.0.97.3:43811 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:48.080175 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.071847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:54635)
0619 00:53:48.072012 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:48.072032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:48.072128 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:48.072448 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:48.072451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:48.072737 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:48.072955 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.072964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.074066 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.074070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.075116 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.075128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.076014 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.076024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.076211 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.076918 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:48.076936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:48.079859 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:53:48.080304 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:48.071899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54635 (local address 127.0.97.62:45087)
0619 00:53:48.072111 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.072116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.072132 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:53:48.072229 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.072232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.072318 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.072401 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.073096 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.073922 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.075292 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.075728 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.075893 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.080049 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54635: BlockingRecv error: recv got EOF from 127.0.97.3:54635 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260619 00:53:48.080603 9651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:48.159687 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.151420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:36807)
0619 00:53:48.151926 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:48.151942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:48.152089 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:48.152435 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:48.152440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:48.152983 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:48.153178 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.153187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.154195 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.154201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.155166 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.155176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.155929 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.155936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.156130 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.156671 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:48.156687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:48.159465 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
W20260619 00:53:48.159890 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:48.151549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36807 (local address 127.0.97.62:45087)
0619 00:53:48.151771 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.151777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.152027 (+ 250us) server_negotiation.cc:407] Connection header received
0619 00:53:48.152219 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.152224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.152296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.152404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.153322 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.154055 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.155309 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.155746 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.155892 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.159664 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36807: BlockingRecv error: recv got EOF from 127.0.97.1:36807 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:48 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:48.624797 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:48.614740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:43803)
0619 00:53:48.615401 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:48.615422 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:48.615574 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:48.616049 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:48.616055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:48.616690 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:48.616981 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.616994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.618626 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.618632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.620277 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:48.620290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.621102 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.621107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:48.621243 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.621899 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:48.621916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:48.624577 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":357,"threads_started":1}
W20260619 00:53:48.625196 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:48.614797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43803 (local address 127.0.97.62:45087)
0619 00:53:48.615001 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:48.615006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:48.615537 (+ 531us) server_negotiation.cc:407] Connection header received
0619 00:53:48.615730 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:48.615737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:48.615884 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:48.615991 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:48.617166 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.618423 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.620439 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:48.620915 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:48.621015 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:48.624873 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43803: BlockingRecv error: recv got EOF from 127.0.97.2:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:49.094671 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:49.081247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:34181)
0619 00:53:49.081785 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:49.081802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:49.081920 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:49.082318 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:49.082323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:49.083010 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:49.083280 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.083292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.084579 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.084583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.085548 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.085560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.086304 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.086310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.086485 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.091254 (+ 4769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:49.091275 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:49.094369 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":308,"threads_started":1}
W20260619 00:53:49.094947 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:49.081371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34181 (local address 127.0.97.62:45087)
0619 00:53:49.081555 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:49.081560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:49.081891 (+ 331us) server_negotiation.cc:407] Connection header received
0619 00:53:49.082034 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:49.082039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:49.082129 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:49.082256 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:49.083483 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.084429 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.085674 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.086087 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.086258 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.094657 (+ 8399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34181: BlockingRecv error: recv got EOF from 127.0.97.3:34181 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:49.169222 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:49.160649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:53257)
0619 00:53:49.161183 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:49.161203 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:49.161366 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:49.161708 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:49.161713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:49.162107 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:49.162248 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.162256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.163329 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.163341 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.164350 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.164361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.165110 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.165117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.165280 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.165817 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:49.165833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:49.168872 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
W20260619 00:53:49.169471 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:49.160731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53257 (local address 127.0.97.62:45087)
0619 00:53:49.161020 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:49.161026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:49.161292 (+ 266us) server_negotiation.cc:407] Connection header received
0619 00:53:49.161488 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:49.161499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:49.161572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:49.161676 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:49.162394 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.163172 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.164491 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.164952 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.165079 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.169065 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53257: BlockingRecv error: recv got EOF from 127.0.97.1:53257 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:49 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:49.633806 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:49.625721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:55263)
0619 00:53:49.626246 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:49.626263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:49.626376 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:49.626674 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:49.626677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:49.627069 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:49.627203 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.627211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.628174 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.628178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.629145 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:49.629157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.629874 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.629879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:49.630002 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.630570 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:49.630587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:49.633522 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
W20260619 00:53:49.634027 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:49.625827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55263 (local address 127.0.97.62:45087)
0619 00:53:49.626003 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:49.626007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:49.626348 (+ 341us) server_negotiation.cc:407] Connection header received
0619 00:53:49.626473 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:49.626477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:49.626548 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:49.626631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:49.627333 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.628034 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.629277 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:49.629696 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:49.629829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:49.633774 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55263: BlockingRecv error: recv got EOF from 127.0.97.2:55263 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:50.104252 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:50.096047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:36695)
0619 00:53:50.096590 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:50.096606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:50.096730 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:50.097088 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:50.097091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:50.097467 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:50.097615 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.097624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.098687 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.098694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.099616 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.099628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.100321 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.100327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.100499 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.101179 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:50.101197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:50.103912 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":293,"threads_started":1}
W20260619 00:53:50.104359 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:50.096258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36695 (local address 127.0.97.62:45087)
0619 00:53:50.096477 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:50.096482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:50.096701 (+ 219us) server_negotiation.cc:407] Connection header received
0619 00:53:50.096894 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:50.096897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:50.096959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:50.097046 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:50.097756 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.098528 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.099760 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.100163 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.100298 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.104111 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36695: BlockingRecv error: recv got EOF from 127.0.97.3:36695 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:50.179795 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:50.170249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:56631)
0619 00:53:50.170749 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:50.170768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:50.170888 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:50.171193 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:50.171196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:50.171608 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:50.171823 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.171833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.173364 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.173372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.174512 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.174576 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.175456 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.175464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.175688 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.176426 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:50.176448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:50.179435 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":288,"threads_started":1}
W20260619 00:53:50.179900 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:50.170345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56631 (local address 127.0.97.62:45087)
0619 00:53:50.170618 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:50.170626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:50.170862 (+ 236us) server_negotiation.cc:407] Connection header received
0619 00:53:50.170983 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:50.170987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:50.171058 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:50.171160 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:50.171996 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.173177 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.174738 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.175237 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.175435 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.179690 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56631: BlockingRecv error: recv got EOF from 127.0.97.1:56631 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:50 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:50.643018 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:50.634886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:43527)
0619 00:53:50.635393 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:50.635411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:50.635530 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:50.635847 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:50.635850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:50.636173 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:50.636375 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.636383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.637523 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.637527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.638397 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:50.638406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.639069 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.639074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:50.639197 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.639848 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:50.639866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:50.642618 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
W20260619 00:53:50.643085 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:50.634958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43527 (local address 127.0.97.62:45087)
0619 00:53:50.635197 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:50.635204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:50.635525 (+ 321us) server_negotiation.cc:407] Connection header received
0619 00:53:50.635636 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:50.635640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:50.635723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:50.635808 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:50.636552 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.637387 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.638537 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:50.638933 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:50.639030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:50.642829 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43527: BlockingRecv error: recv got EOF from 127.0.97.2:43527 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:51.113441 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:51.105201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:55641)
0619 00:53:51.105780 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:51.105796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:51.105913 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:51.106264 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:51.106271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:51.106626 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:51.106785 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.106795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.107769 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.107776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.108666 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.108677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.109429 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.109435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.109607 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.110299 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:51.110331 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:51.113204 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
W20260619 00:53:51.113603 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:51.105365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55641 (local address 127.0.97.62:45087)
0619 00:53:51.105556 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:51.105560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:51.105882 (+ 322us) server_negotiation.cc:407] Connection header received
0619 00:53:51.106057 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:51.106062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:51.106121 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:51.106224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:51.106947 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.107639 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.108836 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.109269 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.109407 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.113408 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55641: BlockingRecv error: recv got EOF from 127.0.97.3:55641 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:51.189401 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:51.180756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:50689)
0619 00:53:51.181300 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:51.181316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:51.181479 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:51.181833 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:51.181838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:51.182252 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:51.182407 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.182416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.183695 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.183702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.184883 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.184895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.185548 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.185551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.185683 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.186245 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:51.186262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:51.189066 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":324,"threads_started":1}
W20260619 00:53:51.189522 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:51.180839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50689 (local address 127.0.97.62:45087)
0619 00:53:51.181051 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:51.181057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:51.181412 (+ 355us) server_negotiation.cc:407] Connection header received
0619 00:53:51.181613 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:51.181617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:51.181679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:51.181834 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:51.182556 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.183508 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.185003 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.185419 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.185516 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.189248 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50689: BlockingRecv error: recv got EOF from 127.0.97.1:50689 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:51 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:51.653295 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:51.644104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:38789)
0619 00:53:51.644630 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:51.644647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:51.644761 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:51.645159 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:51.645163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:51.645541 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:51.645693 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.645700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.646700 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.646703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.647710 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:51.647722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.648580 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.648585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:51.648802 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.649600 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:51.649618 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:51.653051 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
W20260619 00:53:51.653484 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:51.644158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38789 (local address 127.0.97.62:45087)
0619 00:53:51.644344 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:51.644348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:51.644758 (+ 410us) server_negotiation.cc:407] Connection header received
0619 00:53:51.644932 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:51.644935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:51.645021 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:51.645109 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:51.645875 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.646584 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.647864 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:51.648408 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:51.648517 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:51.653226 (+ 4709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38789: BlockingRecv error: recv got EOF from 127.0.97.2:38789 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:52.123097 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:52.114533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:56211)
0619 00:53:52.115006 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:52.115024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:52.115154 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:52.115511 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:52.115515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:52.115917 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:52.116077 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.116086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.117247 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.117252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.118227 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.118239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.119025 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.119029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.119180 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.119923 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:52.119940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:52.122859 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
W20260619 00:53:52.123347 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:52.114694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56211 (local address 127.0.97.62:45087)
0619 00:53:52.114951 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:52.114958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:52.115140 (+ 182us) server_negotiation.cc:407] Connection header received
0619 00:53:52.115259 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:52.115262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:52.115343 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:52.115431 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:52.116241 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.117101 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.118385 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.118881 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.118997 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.123085 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56211: BlockingRecv error: recv got EOF from 127.0.97.3:56211 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:52.199056 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:52.190492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:37879)
0619 00:53:52.191088 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:52.191104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:52.191306 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:52.191661 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:52.191665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:52.192080 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:52.192256 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.192265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.193517 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.193523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.194438 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.194449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.195270 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.195276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.195447 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.196115 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:52.196133 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:52.198825 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
W20260619 00:53:52.199335 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:52.190618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37879 (local address 127.0.97.62:45087)
0619 00:53:52.190905 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:52.190912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:52.191197 (+ 285us) server_negotiation.cc:407] Connection header received
0619 00:53:52.191441 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:52.191446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:52.191523 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:52.191630 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:52.192413 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.193387 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.194583 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.195087 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.195235 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.199040 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37879: BlockingRecv error: recv got EOF from 127.0.97.1:37879 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:52 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:52.663053 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:52.654284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:58431)
0619 00:53:52.654838 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:52.654855 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:52.654969 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:52.655340 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:52.655344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:52.655701 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:52.655866 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.655874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.656952 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.656956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.657886 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:52.657897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.658773 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.658779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:52.658957 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.659736 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:52.659765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:52.662797 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
W20260619 00:53:52.663250 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:52.654579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58431 (local address 127.0.97.62:45087)
0619 00:53:52.654765 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:52.654770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:52.654940 (+ 170us) server_negotiation.cc:407] Connection header received
0619 00:53:52.655059 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:52.655063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:52.655145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:52.655245 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:52.656001 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.656767 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.658041 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:52.658580 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:52.658765 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:52.663017 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58431: BlockingRecv error: recv got EOF from 127.0.97.2:58431 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:53.132289 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:53.124048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:51427)
0619 00:53:53.124582 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:53.124598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:53.124701 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:53.125034 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:53.125038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:53.125401 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:53.125543 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.125551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.126483 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.126487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.127373 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.127383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.128188 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.128194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.128404 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.129083 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:53.129101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:53.131944 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
W20260619 00:53:53.132382 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:53.124249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51427 (local address 127.0.97.62:45087)
0619 00:53:53.124422 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:53.124426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:53.124669 (+ 243us) server_negotiation.cc:407] Connection header received
0619 00:53:53.124796 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:53.124799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:53.124915 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:53.124997 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:53.125655 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.126356 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.127483 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.128045 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.128175 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.132123 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51427: BlockingRecv error: recv got EOF from 127.0.97.3:51427 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:53.209107 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:53.200019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:60729)
0619 00:53:53.200563 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:53.200578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:53.200740 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:53.201194 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:53.201199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:53.201733 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:53.201925 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.201935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.203229 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.203235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.204274 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.204285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.205118 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.205126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.205307 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.205942 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:53.205960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:53.208767 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":324,"threads_started":1}
W20260619 00:53:53.209283 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:53.200159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60729 (local address 127.0.97.62:45087)
0619 00:53:53.200381 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:53.200389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:53.200749 (+ 360us) server_negotiation.cc:407] Connection header received
0619 00:53:53.200901 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:53.200928 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:53.201034 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:53.201163 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:53.202085 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.203086 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.204410 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.204966 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.205100 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.209026 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60729: BlockingRecv error: recv got EOF from 127.0.97.1:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:53 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:53.671672 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:53.663959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:59555)
0619 00:53:53.664435 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:53.664452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:53.664565 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:53.664887 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:53.664890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:53.665250 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:53.665397 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.665404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.666311 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.666314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.667179 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:53.667189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.667930 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.667934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:53.668047 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.668636 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:53.668652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:53.671350 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
W20260619 00:53:53.671823 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:53.664121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59555 (local address 127.0.97.62:45087)
0619 00:53:53.664362 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:53.664368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:53.664532 (+ 164us) server_negotiation.cc:407] Connection header received
0619 00:53:53.664665 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:53.664668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:53.664744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:53.664851 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:53.665519 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.666197 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.667350 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:53.667776 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:53.667875 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:53.671577 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59555: BlockingRecv error: recv got EOF from 127.0.97.2:59555 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:54.145901 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:54.133337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:47143)
0619 00:53:54.133884 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:54.133900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:54.134068 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:54.134404 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:54.134408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:54.134854 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:54.134998 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.135007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.135987 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.135993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.137038 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.137049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.137825 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.137829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.138094 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.142683 (+ 4589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:54.142704 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:54.145670 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
W20260619 00:53:54.146173 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:54.133451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47143 (local address 127.0.97.62:45087)
0619 00:53:54.133647 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:54.133654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:54.133995 (+ 341us) server_negotiation.cc:407] Connection header received
0619 00:53:54.134204 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:54.134208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:54.134268 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:54.134371 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:54.135146 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.135847 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.137176 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.137687 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.137816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.145900 (+ 8084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47143: BlockingRecv error: recv got EOF from 127.0.97.3:47143 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:54.218015 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:54.210127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:51393)
0619 00:53:54.210716 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:54.210732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:54.210848 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:54.211126 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:54.211129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:54.211504 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:54.211667 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.211675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.212650 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.212654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.213635 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.213647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.214325 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.214329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.214441 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.214988 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:54.215005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:54.217782 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":262,"threads_started":1}
W20260619 00:53:54.218283 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:54.210212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51393 (local address 127.0.97.62:45087)
0619 00:53:54.210415 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:54.210424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:54.210822 (+ 398us) server_negotiation.cc:407] Connection header received
0619 00:53:54.210944 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:54.210948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:54.211008 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:54.211086 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:54.211795 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.212506 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.213793 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.214195 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.214290 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.218009 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51393: BlockingRecv error: recv got EOF from 127.0.97.1:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:54 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:54.681144 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:54.672667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:47941)
0619 00:53:54.673305 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:54.673321 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:54.673458 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:54.673782 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:54.673785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:54.674295 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:54.674507 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.674515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.675723 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.675729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.676632 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:54.676644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.677377 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.677383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:54.677545 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.678079 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:54.678096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:54.680896 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":280,"threads_started":1}
W20260619 00:53:54.681452 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:54.672735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47941 (local address 127.0.97.62:45087)
0619 00:53:54.673042 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:54.673049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:54.673415 (+ 366us) server_negotiation.cc:407] Connection header received
0619 00:53:54.673559 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:54.673562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:54.673664 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:54.673746 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:54.674655 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.675585 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.676770 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:54.677223 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:54.677366 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:54.681126 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47941: BlockingRecv error: recv got EOF from 127.0.97.2:47941 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:55.155689 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:55.146778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:36623)
0619 00:53:55.147429 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:55.147445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:55.147557 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:55.147945 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:55.147948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:55.148324 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:55.148516 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.148524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.149683 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.149687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.150726 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.150741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.151824 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.151830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.151999 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.152741 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:55.152763 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:55.155453 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
W20260619 00:53:55.155807 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:55.146859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36623 (local address 127.0.97.62:45087)
0619 00:53:55.147022 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:55.147027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:55.147520 (+ 493us) server_negotiation.cc:407] Connection header received
0619 00:53:55.147719 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:55.147723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:55.147791 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:55.147905 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:55.148669 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.149563 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.150923 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.151633 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.151767 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.155622 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36623: BlockingRecv error: recv got EOF from 127.0.97.3:36623 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:55.227216 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:55.218948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:43723)
0619 00:53:55.219458 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:55.219475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:55.219589 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:55.219869 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:55.219872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:55.220232 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:55.220381 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.220388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.221373 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.221376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.222270 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.222281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.223205 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.223209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.223398 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.224084 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:55.224101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:55.226856 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
W20260619 00:53:55.227281 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:55.219160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43723 (local address 127.0.97.62:45087)
0619 00:53:55.219370 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:55.219376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:55.219554 (+ 178us) server_negotiation.cc:407] Connection header received
0619 00:53:55.219670 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:55.219674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:55.219732 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:55.219817 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:55.220500 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.221246 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.222474 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.223037 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.223183 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.227052 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43723: BlockingRecv error: recv got EOF from 127.0.97.1:43723 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:55 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:55.690774 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:55.682072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:49557)
0619 00:53:55.682638 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:55.682654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:55.682761 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:55.683088 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:55.683091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:55.683485 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:55.683624 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.683632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.684753 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.684758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.685785 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:55.685797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.686540 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.686544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:55.686662 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.687236 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:55.687253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:55.690501 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":306,"threads_started":1}
W20260619 00:53:55.690945 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:55.682122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49557 (local address 127.0.97.62:45087)
0619 00:53:55.682355 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:55.682361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:55.682725 (+ 364us) server_negotiation.cc:407] Connection header received
0619 00:53:55.682856 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:55.682859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:55.682957 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:55.683035 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:55.683749 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.684596 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.685934 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:55.686372 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:55.686475 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:55.690693 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49557: BlockingRecv error: recv got EOF from 127.0.97.2:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:56.164781 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:56.156611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:52411)
0619 00:53:56.157147 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:56.157163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:56.157278 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:56.157580 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:56.157583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:56.157959 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:56.158110 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.158118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.159042 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.159045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.159962 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.159972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.160676 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.160680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.160794 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.161634 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:56.161651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:56.164561 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":316,"threads_started":1}
W20260619 00:53:56.165076 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:56.156708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52411 (local address 127.0.97.62:45087)
0619 00:53:56.156936 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:56.156941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:56.157237 (+ 296us) server_negotiation.cc:407] Connection header received
0619 00:53:56.157370 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:56.157373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:56.157437 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:56.157520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:56.158231 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.158910 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.160096 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.160517 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.160612 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.164749 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52411: BlockingRecv error: recv got EOF from 127.0.97.3:52411 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:56.237294 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:56.228286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:47841)
0619 00:53:56.228894 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:56.228910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:56.229077 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:56.229430 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:56.229435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:56.229862 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:56.230041 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.230050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.231107 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.231113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.232062 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.232074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.232858 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.232865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.233055 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.233709 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:56.233727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:56.236938 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":340,"threads_started":1}
W20260619 00:53:56.237424 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:56.228417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47841 (local address 127.0.97.62:45087)
0619 00:53:56.228680 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:56.228689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:56.229007 (+ 318us) server_negotiation.cc:407] Connection header received
0619 00:53:56.229211 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:56.229217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:56.229287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:56.229429 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:56.230207 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.230963 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.232213 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.232655 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.232800 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.237167 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47841: BlockingRecv error: recv got EOF from 127.0.97.1:47841 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:56 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:56.700712 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:56.691733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:43847)
0619 00:53:56.692292 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:56.692309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:56.692489 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:56.692938 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:56.692943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:56.693395 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:56.693560 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.693569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.694653 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.694661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.695864 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:56.695878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.696716 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.696721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:56.696905 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.697534 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:56.697552 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:56.700407 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
W20260619 00:53:56.700874 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:56.691834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43847 (local address 127.0.97.62:45087)
0619 00:53:56.692104 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:56.692113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:56.692407 (+ 294us) server_negotiation.cc:407] Connection header received
0619 00:53:56.692643 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:56.692648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:56.692748 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:56.692917 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:56.693732 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.694530 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.696047 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:56.696569 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:56.696669 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:56.700630 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43847: BlockingRecv error: recv got EOF from 127.0.97.2:43847 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:57.174801 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:57.165860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:41565)
0619 00:53:57.166353 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:57.166369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:57.166490 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:57.166850 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:57.166854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:57.167304 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:57.167508 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.167519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.168908 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.168915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.169952 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.169966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.170673 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.170679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.170866 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.171463 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:57.171480 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:57.174415 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":291,"threads_started":1}
W20260619 00:53:57.174844 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:57.166064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41565 (local address 127.0.97.62:45087)
0619 00:53:57.166296 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:57.166303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:57.166453 (+ 150us) server_negotiation.cc:407] Connection header received
0619 00:53:57.166590 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:57.166594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:57.166654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:57.166737 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:57.167701 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.168728 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.170104 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.170516 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.170657 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.174635 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41565: BlockingRecv error: recv got EOF from 127.0.97.3:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:57.247274 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:57.238523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:45539)
0619 00:53:57.239078 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:57.239094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:57.239261 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:57.239629 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:57.239636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:57.240030 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:57.240174 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.240183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.241307 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.241313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.242252 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.242263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.242993 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.242998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.243175 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.243871 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:57.243893 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:57.246929 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":340,"threads_started":1}
W20260619 00:53:57.247455 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:57.238636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45539 (local address 127.0.97.62:45087)
0619 00:53:57.238931 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:57.238937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:57.239187 (+ 250us) server_negotiation.cc:407] Connection header received
0619 00:53:57.239393 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:57.239400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:57.239468 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:57.239576 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:57.240330 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.241163 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.242399 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.242821 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.242982 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.247119 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45539: BlockingRecv error: recv got EOF from 127.0.97.1:45539 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:57 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:57.709842 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:57.701656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:46809)
0619 00:53:57.702217 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:57.702237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:57.702381 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:57.702683 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:57.702686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:57.703049 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:57.703189 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.703197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.704159 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.704163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.705237 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:57.705250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.706024 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.706028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:57.706155 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.706778 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:57.706795 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:57.709597 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":319,"threads_started":1}
W20260619 00:53:57.710036 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:57.701717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46809 (local address 127.0.97.62:45087)
0619 00:53:57.701947 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:57.701956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:57.702320 (+ 364us) server_negotiation.cc:407] Connection header received
0619 00:53:57.702495 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:57.702499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:57.702569 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:57.702652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:57.703322 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.704030 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.705440 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:57.705863 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:57.705970 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:57.709812 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46809: BlockingRecv error: recv got EOF from 127.0.97.2:46809 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:58.184068 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:58.175759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:54963)
0619 00:53:58.176274 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:58.176290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:58.176405 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:58.176698 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:58.176701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:58.177080 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:58.177221 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.177229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.178263 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.178267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.179192 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.179202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.179931 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.179935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.180050 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.180607 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:58.180623 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:58.183850 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":305,"threads_started":1}
W20260619 00:53:58.184303 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:58.175936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54963 (local address 127.0.97.62:45087)
0619 00:53:58.176100 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:58.176104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:58.176370 (+ 266us) server_negotiation.cc:407] Connection header received
0619 00:53:58.176509 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:58.176512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:58.176576 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:58.176659 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:58.177371 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.178131 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.179319 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.179788 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.179878 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.184052 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54963: BlockingRecv error: recv got EOF from 127.0.97.3:54963 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:58.257609 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:58.248388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:58273)
0619 00:53:58.248916 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:58.248933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:58.249046 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:58.249382 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:58.249386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:58.249900 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:58.250096 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.250105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.251209 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.251213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.252168 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.252181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.253006 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.253011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.253130 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.253826 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:58.253863 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:58.257248 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260619 00:53:58.257822 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:58.248506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58273 (local address 127.0.97.62:45087)
0619 00:53:58.248763 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:58.248769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:58.249007 (+ 238us) server_negotiation.cc:407] Connection header received
0619 00:53:58.249154 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:58.249158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:58.249218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:58.249299 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:58.250250 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.251029 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.252339 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.252776 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.252946 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.257534 (+ 4588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58273: BlockingRecv error: recv got EOF from 127.0.97.1:58273 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:58 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:58.719740 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:58.710802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:34169)
0619 00:53:58.711339 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:58.711354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:58.711480 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:58.711895 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:58.711899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:58.712276 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:58.712481 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.712490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.713612 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.713617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.714765 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:58.714779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.715672 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.715679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:58.715848 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.716503 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:58.716521 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:58.719499 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":309,"threads_started":1}
W20260619 00:53:58.720007 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:58.710956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34169 (local address 127.0.97.62:45087)
0619 00:53:58.711125 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:58.711129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:58.711449 (+ 320us) server_negotiation.cc:407] Connection header received
0619 00:53:58.711650 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:58.711654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:58.711757 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:58.711852 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:58.712644 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.713435 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.714903 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:58.715502 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:58.715659 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:58.719738 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34169: BlockingRecv error: recv got EOF from 127.0.97.2:34169 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:59.197826 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:59.185024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:33095)
0619 00:53:59.185585 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:59.185601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:59.185714 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:59.186016 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:59.186020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:59.186418 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:59.186585 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.186594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.187560 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.187564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.188447 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.188458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.189349 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.189354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.189584 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.194687 (+ 5103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:59.194705 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:59.197585 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":333,"threads_started":1}
W20260619 00:53:59.198001 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:59.185069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33095 (local address 127.0.97.62:45087)
0619 00:53:59.185225 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:59.185230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:59.185715 (+ 485us) server_negotiation.cc:407] Connection header received
0619 00:53:59.185793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:59.185798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:59.185863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:59.185957 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:59.186720 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.187441 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.188592 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.189156 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.189333 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.197795 (+ 8462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33095: BlockingRecv error: recv got EOF from 127.0.97.3:33095 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:59.267050 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:59.258637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:35507)
0619 00:53:59.259178 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:59.259194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:59.259310 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:59.259604 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:59.259607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:59.259968 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:59.260115 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.260123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.261223 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.261231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.262384 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.262396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.263116 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.263123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.263301 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.263887 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:59.263905 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:59.266724 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":322,"threads_started":1}
W20260619 00:53:59.267158 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:59.258707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35507 (local address 127.0.97.62:45087)
0619 00:53:59.258903 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:59.258908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:59.259306 (+ 398us) server_negotiation.cc:407] Connection header received
0619 00:53:59.259398 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:59.259402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:59.259464 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:59.259555 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:59.260248 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.261051 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.262519 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.262950 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.263092 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.266927 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35507: BlockingRecv error: recv got EOF from 127.0.97.1:35507 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:53:59 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:53:59.729669 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:53:59.720941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:47233)
0619 00:53:59.721483 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 00:53:59.721498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:53:59.721608 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:53:59.721883 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:53:59.721886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:53:59.722245 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:53:59.722422 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.722431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.723805 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.723812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.725145 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:53:59.725157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.725814 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.725818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:53:59.725955 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.726598 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:53:59.726616 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:53:59.729447 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":303,"threads_started":1}
W20260619 00:53:59.729851 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:53:59.720959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47233 (local address 127.0.97.62:45087)
0619 00:53:59.721138 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 00:53:59.721142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:53:59.721575 (+ 433us) server_negotiation.cc:407] Connection header received
0619 00:53:59.721706 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:53:59.721710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:53:59.721771 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:53:59.721860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:53:59.722544 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.723625 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.725265 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:53:59.725677 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:53:59.725778 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:53:59.729646 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47233: BlockingRecv error: recv got EOF from 127.0.97.2:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:00.206961 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:00.198853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:60145)
0619 00:54:00.199382 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:00.199398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:00.199509 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:00.199833 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:00.199836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:00.200223 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:00.200380 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.200391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.201364 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.201368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.202293 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.202304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.202975 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.202989 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.203117 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.203797 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:00.203814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:00.206728 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":305,"threads_started":1}
W20260619 00:54:00.207315 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:00.198911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60145 (local address 127.0.97.62:45087)
0619 00:54:00.199065 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:00.199070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:00.199501 (+ 431us) server_negotiation.cc:407] Connection header received
0619 00:54:00.199610 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:00.199613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:00.199676 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:00.199767 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:00.200512 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.201228 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.202424 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.202845 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.202940 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.206954 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60145: BlockingRecv error: recv got EOF from 127.0.97.3:60145 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:00.277463 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:00.268251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:37937)
0619 00:54:00.268853 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:00.268870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:00.269044 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:00.269401 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:00.269406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:00.269839 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:00.270031 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.270040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.271147 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.271152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.272574 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.272595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.273465 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.273470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.273585 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.274280 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:00.274312 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:00.277125 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":290,"threads_started":1}
W20260619 00:54:00.277559 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:00.268334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37937 (local address 127.0.97.62:45087)
0619 00:54:00.268629 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:00.268637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:00.268974 (+ 337us) server_negotiation.cc:407] Connection header received
0619 00:54:00.269178 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:00.269189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:00.269251 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:00.269367 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:00.270189 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.270987 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.272713 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.273293 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.273402 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.277316 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37937: BlockingRecv error: recv got EOF from 127.0.97.1:37937 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:00 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:00.738838 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:00.730770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:33805)
0619 00:54:00.731328 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:00.731345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:00.731482 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:00.731746 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:00.731749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:00.732122 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:00.732297 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.732305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.733392 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.733397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.734406 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:00.734419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.735101 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.735105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:00.735219 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.735832 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:00.735848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:00.738605 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":297,"threads_started":1}
W20260619 00:54:00.739030 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:00.730889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33805 (local address 127.0.97.62:45087)
0619 00:54:00.731050 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:00.731056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:00.731449 (+ 393us) server_negotiation.cc:407] Connection header received
0619 00:54:00.731562 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:00.731565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:00.731622 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:00.731701 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:00.732418 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.733244 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.734545 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:00.734938 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:00.735027 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:00.738785 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33805: BlockingRecv error: recv got EOF from 127.0.97.2:33805 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:01.216049 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:01.208003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:43491)
0619 00:54:01.208587 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:01.208603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:01.208714 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:01.209014 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:01.209016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:01.209391 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:01.209550 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.209559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.210528 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.210532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.211449 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.211460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.212177 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.212181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.212299 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.212991 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:01.213010 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:01.215805 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":343,"threads_started":1}
W20260619 00:54:01.216307 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:01.208032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43491 (local address 127.0.97.62:45087)
0619 00:54:01.208258 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:01.208267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:01.208683 (+ 416us) server_negotiation.cc:407] Connection header received
0619 00:54:01.208848 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:01.208852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:01.208919 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:01.209000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:01.209680 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.210381 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.211585 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.212018 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.212112 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.216041 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43491: BlockingRecv error: recv got EOF from 127.0.97.3:43491 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:01.287439 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:01.278588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:51801)
0619 00:54:01.279093 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:01.279112 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:01.279275 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:01.279619 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:01.279645 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:01.280155 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:01.280357 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.280369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.281816 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.281820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.282819 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.282830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.283585 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.283591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.283770 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.284355 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:01.284372 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:01.287084 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":273,"threads_started":1}
W20260619 00:54:01.287535 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:01.278697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51801 (local address 127.0.97.62:45087)
0619 00:54:01.279010 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:01.279016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:01.279221 (+ 205us) server_negotiation.cc:407] Connection header received
0619 00:54:01.279416 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:01.279421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:01.279488 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:01.279598 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:01.280571 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.281683 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.282952 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.283393 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.283573 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.287281 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51801: BlockingRecv error: recv got EOF from 127.0.97.1:51801 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:01 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:01.748487 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:01.739821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:60961)
0619 00:54:01.740334 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:01.740352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:01.740474 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:01.740763 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:01.740766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:01.741194 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:01.741345 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.741353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.742485 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.742490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.743452 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:01.743464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.744248 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.744252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:01.744387 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.745073 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:01.745092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:01.748136 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":269,"threads_started":1}
W20260619 00:54:01.748636 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:01.740017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60961 (local address 127.0.97.62:45087)
0619 00:54:01.740232 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:01.740238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:01.740478 (+ 240us) server_negotiation.cc:407] Connection header received
0619 00:54:01.740565 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:01.740569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:01.740632 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:01.740721 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:01.741560 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.742314 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.743633 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:01.744090 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:01.744204 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:01.748409 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60961: BlockingRecv error: recv got EOF from 127.0.97.2:60961 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:02.226034 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:02.217255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.3:36355)
0619 00:54:02.217858 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:02.217875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:02.218057 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:02.218479 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:02.218485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:02.218874 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:02.219048 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.219057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.220087 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.220093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.221073 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.221086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.221842 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.221848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.222029 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.222721 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:02.222739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:02.225675 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":361,"threads_started":1}
W20260619 00:54:02.226163 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:02.217424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36355 (local address 127.0.97.62:45087)
0619 00:54:02.217618 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:02.217624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:02.217993 (+ 369us) server_negotiation.cc:407] Connection header received
0619 00:54:02.218230 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:02.218234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:02.218299 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:02.218411 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:02.219202 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.219923 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.221228 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.221680 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.221817 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.225911 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36355: BlockingRecv error: recv got EOF from 127.0.97.3:36355 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:02.298058 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:02.288799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.1:52335)
0619 00:54:02.289529 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:02.289549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:02.289691 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:02.290212 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:02.290216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:02.290687 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:02.290923 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.290936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.292153 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.292159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.293177 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.293190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.293901 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.293907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.294084 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.294775 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:02.294793 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:02.297687 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":433,"threads_started":1}
W20260619 00:54:02.298148 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:02.288958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52335 (local address 127.0.97.62:45087)
0619 00:54:02.289180 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:02.289185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:02.289676 (+ 491us) server_negotiation.cc:407] Connection header received
0619 00:54:02.289852 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:02.289856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:02.289927 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:02.290029 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:02.291121 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.291995 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.293322 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.293737 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.293885 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.297919 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52335: BlockingRecv error: recv got EOF from 127.0.97.1:52335 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 00:54:02 dist-test-slave-r7bh krb5kdc[8654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 00:54:02.759467 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:02.749875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45087 (local address 127.0.97.2:35617)
0619 00:54:02.750476 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:02.750493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:02.750668 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:02.751280 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:02.751285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:02.751900 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:02.752081 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.752091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.753229 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.753233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.754182 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:02.754194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.755024 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.755028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:02.755157 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.755780 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:02.755798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:02.758940 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45087: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":322,"threads_started":1}
W20260619 00:54:02.759534 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:02.749924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35617 (local address 127.0.97.62:45087)
0619 00:54:02.750209 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:02.750218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:02.750591 (+ 373us) server_negotiation.cc:407] Connection header received
0619 00:54:02.750968 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:02.750976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:02.751087 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:02.751244 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:02.752258 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.753051 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.754383 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:02.754836 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:02.754947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:02.759292 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35617: BlockingRecv error: recv got EOF from 127.0.97.2:35617 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 00:54:03.046401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9241
I20260619 00:54:03.060101 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:03.047500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.97.1:36081)
0619 00:54:03.048036 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:03.048044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:03.048073 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:54:03.048134 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:03.048140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:03.048381 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:03.048540 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:03.050155 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.051390 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.053089 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.053737 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.053857 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:03.056767 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:03.056778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:03.056802 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:03.056926 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:03.059589 (+ 2663us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 00:54:03.059876 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40190: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
W20260619 00:54:03.060314 9701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40190: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 00:54:03.060552 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9241: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36081: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 00:54:03.060789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9241
I20260619 00:54:03.077770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9392
I20260619 00:54:03.090590 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:03.078784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.97.2:36559)
0619 00:54:03.079406 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:03.079414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:03.079466 (+ 52us) server_negotiation.cc:407] Connection header received
0619 00:54:03.079533 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:03.079539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:03.079820 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:03.079984 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:03.081518 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.082599 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.083988 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.084596 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.084705 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:03.087216 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:03.087225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:03.087239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:03.087273 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:03.090037 (+ 2764us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 00:54:03.090312 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52570: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
W20260619 00:54:03.090827 9702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52570: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 00:54:03.090819 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9392: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36559: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 00:54:03.091018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9392
I20260619 00:54:03.108059 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9523
I20260619 00:54:03.120436 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:03.109201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.97.3:40403)
0619 00:54:03.109835 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:03.109843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:03.109871 (+ 28us) server_negotiation.cc:407] Connection header received
0619 00:54:03.109941 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:03.109947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:03.110168 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:03.110320 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:03.111617 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.112666 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.114002 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.114583 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.114697 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:03.117201 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:03.117210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:03.117225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:03.117258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:03.119919 (+ 2661us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 00:54:03.120189 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34334: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":328,"threads_started":1}
W20260619 00:54:03.120569 9703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34334: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 00:54:03.120647 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9523: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:40403: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 00:54:03.120870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9523
I20260619 00:54:03.137040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9172
I20260619 00:54:03.138329 9234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:03.253994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9172
2026-06-19T00:54:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21106 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T00:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:54:03Z Disabled control of system clock
I20260619 00:54:03.334336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36763
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 00:54:03.682739 9708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:03.683080 9708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:03.683162 9708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:03.690598 9708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:54:03.690734 9708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:03.690774 9708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:03.690825 9708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:54:03.690867 9708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:54:03.700205 9708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:03.701591 9708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:03.703387 9708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:03.716125 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:03.716405 9708 server_base.cc:1034] running on GCE node
W20260619 00:54:03.716079 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:03.716125 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:03.717598 9708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:03.719193 9708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:03.720407 9708 hybrid_clock.cc:648] HybridClock initialized: now 1781830443720361 us; error 33 us; skew 500 ppm
I20260619 00:54:03.720841 9708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:03.723318 9708 webserver.cc:460] Webserver started at http://127.0.97.62:32979/ using document root <none> and password file <none>
I20260619 00:54:03.724175 9708 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:03.724298 9708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:03.724663 9708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:03.727715 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "9e2ea901c84040c0acbdd3514636a409"
format_stamp: "Formatted at 2026-06-19 00:54:03 on dist-test-slave-r7bh"
server_key: "25bdd29fb10dc790a82477b098eae70c"
server_key_iv: "dc48e01c3b28659ab29b16a103f6a584"
server_key_version: "encryptionkey@0"
I20260619 00:54:03.728586 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e2ea901c84040c0acbdd3514636a409"
format_stamp: "Formatted at 2026-06-19 00:54:03 on dist-test-slave-r7bh"
server_key: "25bdd29fb10dc790a82477b098eae70c"
server_key_iv: "dc48e01c3b28659ab29b16a103f6a584"
server_key_version: "encryptionkey@0"
I20260619 00:54:03.734294 9708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 00:54:03.738495 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:03.740214 9708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:54:03.740424 9708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "9e2ea901c84040c0acbdd3514636a409"
format_stamp: "Formatted at 2026-06-19 00:54:03 on dist-test-slave-r7bh"
server_key: "25bdd29fb10dc790a82477b098eae70c"
server_key_iv: "dc48e01c3b28659ab29b16a103f6a584"
server_key_version: "encryptionkey@0"
I20260619 00:54:03.740664 9708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:03.763018 9708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:03.763959 9708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:03.764269 9708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:03.783835 9708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36763
I20260619 00:54:03.783854 9773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36763 every 8 connection(s)
I20260619 00:54:03.785355 9708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:54:03.788312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9708
I20260619 00:54:03.788528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:54:03.789256 388 external_mini_cluster.cc:1428] Setting key 0f97f8b59b27edba820e5d9ab2c0cd26
I20260619 00:54:03.789719 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:03.803848 9774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409: Bootstrap starting.
I20260619 00:54:03.804917 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:03.791235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.97.62:36763)
0619 00:54:03.792426 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:03.792445 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:03.792562 (+ 117us) server_negotiation.cc:407] Connection header received
0619 00:54:03.795053 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:03.795081 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:03.795780 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:03.796246 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:03.796952 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.798289 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.799690 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:03.800439 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:03.800666 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:03.802116 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:03.802125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:03.802144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:03.802772 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:03.802845 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:03.802870 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:03.803038 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:03.803300 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 00:54:03.803704 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":393,"threads_started":1}
I20260619 00:54:03.807582 9774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:03.808765 9774 log.cc:824] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:03.811522 9774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409: No bootstrap required, opened a new log
I20260619 00:54:03.819717 9774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 00:54:03.820154 9774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:03.820271 9774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2ea901c84040c0acbdd3514636a409, State: Initialized, Role: FOLLOWER
I20260619 00:54:03.820933 9774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 00:54:03.821166 9774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:54:03.821308 9774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:54:03.821487 9774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:03.823657 9774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 00:54:03.824168 9774 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2ea901c84040c0acbdd3514636a409; no voters:
I20260619 00:54:03.824862 9774 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:54:03.825112 9779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:54:03.825942 9779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2ea901c84040c0acbdd3514636a409, State: Running, Role: LEADER
I20260619 00:54:03.826576 9779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } }
I20260619 00:54:03.827257 9774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:54:03.830087 9781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2ea901c84040c0acbdd3514636a409. Latest consensus state: current_term: 1 leader_uuid: "9e2ea901c84040c0acbdd3514636a409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } } }
I20260619 00:54:03.830626 9781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:03.831096 9780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2ea901c84040c0acbdd3514636a409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2ea901c84040c0acbdd3514636a409" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36763 } } }
I20260619 00:54:03.831724 9780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:03.832497 9788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:54:03.838935 9788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:54:03.849663 9788 catalog_manager.cc:1347] Generated new cluster ID: 87d7f342a50e4eed8ec75d4cdb773614
I20260619 00:54:03.849825 9788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:54:03.888628 9788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:54:03.890493 9788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:54:03.910876 9788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2ea901c84040c0acbdd3514636a409: Generated new TSK 0
I20260619 00:54:03.912065 9788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:54:03.925066 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 00:54:04.284014 9798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:04.284368 9798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:04.284464 9798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:04.292438 9798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:04.292548 9798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:04.292654 9798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:54:04.301978 9798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:04.303392 9798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:04.305815 9798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:04.321249 9803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:04.321461 9804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:04.322227 9806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:04.323143 9798 server_base.cc:1034] running on GCE node
I20260619 00:54:04.323812 9798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:04.324802 9798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:04.326098 9798 hybrid_clock.cc:648] HybridClock initialized: now 1781830444326024 us; error 63 us; skew 500 ppm
I20260619 00:54:04.326457 9798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:04.333151 9798 webserver.cc:460] Webserver started at http://127.0.97.1:46095/ using document root <none> and password file <none>
I20260619 00:54:04.334017 9798 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:04.334148 9798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:04.334523 9798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:04.337726 9798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "15bb20cf9e744ce0bbff757c33538f33"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "2e79db1b383b70d3420d546d76a56a96"
server_key_iv: "65f10affea4827068173c0a1f97f0042"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.338662 9798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "15bb20cf9e744ce0bbff757c33538f33"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "2e79db1b383b70d3420d546d76a56a96"
server_key_iv: "65f10affea4827068173c0a1f97f0042"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.344367 9798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:54:04.348310 9812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.350155 9798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:54:04.350482 9798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "15bb20cf9e744ce0bbff757c33538f33"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "2e79db1b383b70d3420d546d76a56a96"
server_key_iv: "65f10affea4827068173c0a1f97f0042"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.350739 9798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:04.366070 9798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:04.367043 9798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:04.367360 9798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:04.368384 9798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:04.370208 9798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:04.370350 9798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.370502 9798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:04.370594 9798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.405045 9798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44191
I20260619 00:54:04.405102 9924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44191 every 8 connection(s)
I20260619 00:54:04.406409 9798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:54:04.410285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9798
I20260619 00:54:04.410528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:54:04.411355 388 external_mini_cluster.cc:1428] Setting key 0453f13112115af968277e475c8f40bc
I20260619 00:54:04.414235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 00:54:04.419880 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:04.409195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38743 (local address 127.0.97.62:36763)
0619 00:54:04.409790 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:04.409797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:04.411249 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 00:54:04.413156 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:04.413162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:04.413254 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:04.413378 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:04.414663 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.415383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.417042 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.417447 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.417584 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:04.418998 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:04.419003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:04.419007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:04.419026 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:04.419066 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:04.419078 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:04.419177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:04.419409 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 00:54:04.419506 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
I20260619 00:54:04.421015 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:04.409918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.1:38743)
0619 00:54:04.411057 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:04.411112 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:04.412873 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:04.413655 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:04.413669 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:04.413805 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:04.414486 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:04.414511 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.415536 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.415542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:04.416875 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:04.416895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.417687 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.417696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:04.417937 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:04.418667 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:04.418670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:04.418713 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:04.418793 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:04.419176 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:04.419188 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:04.419199 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:54:04.419403 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 00:54:04.419822 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260619 00:54:04.422796 9925 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 00:54:04.423233 9925 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:04.424080 9925 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
I20260619 00:54:04.426609 9739 ts_manager.cc:194] Registered new tserver with Master: 15bb20cf9e744ce0bbff757c33538f33 (127.0.97.1:44191)
I20260619 00:54:04.429173 9739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:38743
W20260619 00:54:04.756017 9929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:04.756327 9929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:04.756417 9929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:04.763769 9929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:04.763891 9929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:04.763998 9929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:54:04.777606 9929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:04.779011 9929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:04.780754 9929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:04.793947 9934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:04.793917 9935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:04.794600 9937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:04.795068 9929 server_base.cc:1034] running on GCE node
I20260619 00:54:04.795747 9929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:04.796749 9929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:04.798024 9929 hybrid_clock.cc:648] HybridClock initialized: now 1781830444797960 us; error 58 us; skew 500 ppm
I20260619 00:54:04.798358 9929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:04.800729 9929 webserver.cc:460] Webserver started at http://127.0.97.2:35913/ using document root <none> and password file <none>
I20260619 00:54:04.801468 9929 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:04.801600 9929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:04.801934 9929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:04.804864 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "405c2babe6df4615a33d6fb69af865a8"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "0b78e3ee19a601e909ca8c3fc0b8cc0b"
server_key_iv: "87fa6198ec1dbeba4b1eb541d26036b5"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.805788 9929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "405c2babe6df4615a33d6fb69af865a8"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "0b78e3ee19a601e909ca8c3fc0b8cc0b"
server_key_iv: "87fa6198ec1dbeba4b1eb541d26036b5"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.811287 9929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:54:04.815121 9943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.816704 9929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:54:04.816960 9929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "405c2babe6df4615a33d6fb69af865a8"
format_stamp: "Formatted at 2026-06-19 00:54:04 on dist-test-slave-r7bh"
server_key: "0b78e3ee19a601e909ca8c3fc0b8cc0b"
server_key_iv: "87fa6198ec1dbeba4b1eb541d26036b5"
server_key_version: "encryptionkey@0"
I20260619 00:54:04.817207 9929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:04.835038 9929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:04.835992 9929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:04.836320 9929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:04.837370 9929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:04.838972 9929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:04.839095 9929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.839234 9929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:04.839322 9929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:04.876291 9929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36931
I20260619 00:54:04.876343 10055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36931 every 8 connection(s)
I20260619 00:54:04.877801 9929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:54:04.880932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 9929
I20260619 00:54:04.881093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:54:04.881678 388 external_mini_cluster.cc:1428] Setting key 2152c9c4338c2bc323e0a615ea92e621
I20260619 00:54:04.884672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--builtin_ntp_servers=127.0.97.20:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 00:54:04.891765 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:04.880520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32787 (local address 127.0.97.62:36763)
0619 00:54:04.880693 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:04.880697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:04.882676 (+ 1979us) server_negotiation.cc:407] Connection header received
0619 00:54:04.884857 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:04.884863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:04.884980 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:04.885106 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:04.886321 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.887056 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.888744 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.889225 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.889389 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:04.890971 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:04.890976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:04.890979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:04.891004 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:04.891037 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:04.891049 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:04.891146 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:04.891441 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 00:54:04.891527 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 00:54:04.893446 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:04.881282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.2:32787)
0619 00:54:04.882507 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:04.882564 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:04.884525 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:04.885379 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:04.885396 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:04.885503 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:04.886128 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:04.886146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.887232 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.887242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:04.888566 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:04.888579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:04.889503 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:04.889512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:04.889767 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:04.890576 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:04.890579 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:04.890624 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:04.890713 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:04.891160 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:04.891175 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:04.891186 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:54:04.891409 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 00:54:04.891889 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":406,"threads_started":1}
I20260619 00:54:04.895089 10056 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 00:54:04.895437 10056 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:04.896481 10056 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
I20260619 00:54:04.898463 9739 ts_manager.cc:194] Registered new tserver with Master: 405c2babe6df4615a33d6fb69af865a8 (127.0.97.2:36931)
I20260619 00:54:04.899304 9739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:32787
W20260619 00:54:05.245952 10059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:05.246346 10059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:05.246531 10059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:05.255781 10059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:05.255945 10059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:05.256141 10059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:54:05.266911 10059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:05.268456 10059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:05.270661 10059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:05.286012 10064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:05.285979 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:05.285962 10065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:05.287302 10059 server_base.cc:1034] running on GCE node
I20260619 00:54:05.288039 10059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:05.289183 10059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:05.290416 10059 hybrid_clock.cc:648] HybridClock initialized: now 1781830445290381 us; error 36 us; skew 500 ppm
I20260619 00:54:05.290825 10059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:05.293668 10059 webserver.cc:460] Webserver started at http://127.0.97.3:33125/ using document root <none> and password file <none>
I20260619 00:54:05.294464 10059 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:05.294559 10059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:05.294952 10059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:05.298043 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "b7998c2c7d3344b2af27ad259f372f9b"
format_stamp: "Formatted at 2026-06-19 00:54:05 on dist-test-slave-r7bh"
server_key: "d525e60c888535c0f0bb6d89f45404f6"
server_key_iv: "18d4d9a93f512c727fb9b017969d446d"
server_key_version: "encryptionkey@0"
I20260619 00:54:05.298867 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b7998c2c7d3344b2af27ad259f372f9b"
format_stamp: "Formatted at 2026-06-19 00:54:05 on dist-test-slave-r7bh"
server_key: "d525e60c888535c0f0bb6d89f45404f6"
server_key_iv: "18d4d9a93f512c727fb9b017969d446d"
server_key_version: "encryptionkey@0"
I20260619 00:54:05.304502 10059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:54:05.308511 10073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:05.310241 10059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:54:05.310431 10059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "b7998c2c7d3344b2af27ad259f372f9b"
format_stamp: "Formatted at 2026-06-19 00:54:05 on dist-test-slave-r7bh"
server_key: "d525e60c888535c0f0bb6d89f45404f6"
server_key_iv: "18d4d9a93f512c727fb9b017969d446d"
server_key_version: "encryptionkey@0"
I20260619 00:54:05.310703 10059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:05.329102 10059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:05.330101 10059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:05.330427 10059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:05.331671 10059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:05.333293 10059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:05.333382 10059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:05.333535 10059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:05.333635 10059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:05.370267 10059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40033
I20260619 00:54:05.370324 10185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40033 every 8 connection(s)
I20260619 00:54:05.371671 10059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:54:05.381867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10059
I20260619 00:54:05.382042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:54:05.382586 388 external_mini_cluster.cc:1428] Setting key ff0fcc26a2af1feada9147a3de7e2edc
I20260619 00:54:05.387130 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:05.374523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39037 (local address 127.0.97.62:36763)
0619 00:54:05.374742 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:05.374749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:05.376332 (+ 1583us) server_negotiation.cc:407] Connection header received
0619 00:54:05.378426 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:05.378430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:05.378505 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:05.378609 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:05.379929 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.380759 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.383384 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.384014 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.384189 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:05.386184 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:05.386189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:05.386193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:05.386212 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:05.386244 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:05.386257 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:05.386348 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:05.386674 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 00:54:05.386798 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:54:05.388285 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:05.375090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36763 (local address 127.0.97.3:39037)
0619 00:54:05.376140 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:05.376189 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:05.378081 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:05.378936 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:05.378946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:05.379066 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:05.379714 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:05.379732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.381031 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.381036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:05.383164 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:05.383196 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.384308 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.384316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:05.384705 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:05.385760 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:05.385764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:05.385827 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:05.385946 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:05.386394 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:05.386407 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:05.386416 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 00:54:05.386656 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 00:54:05.387071 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260619 00:54:05.390470 10186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36763
I20260619 00:54:05.390923 10186 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:05.391855 10186 heartbeater.cc:511] Master 127.0.97.62:36763 requested a full tablet report, sending...
I20260619 00:54:05.393954 9739 ts_manager.cc:194] Registered new tserver with Master: b7998c2c7d3344b2af27ad259f372f9b (127.0.97.3:40033)
I20260619 00:54:05.394968 9739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39037
I20260619 00:54:05.399614 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 00:54:05.406455 9927 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:05.404612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.0.97.62:36763)
0619 00:54:05.404888 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:05.404898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:05.405309 (+ 411us) server_negotiation.cc:407] Connection header received
0619 00:54:05.405607 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:05.405612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:05.405695 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:05.405809 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:05.406185 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40688: BlockingRecv error: recv got EOF from 127.0.0.1:40688 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:54:05.407402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9798
I20260619 00:54:05.418354 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:05.408457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.0.97.1:44191)
0619 00:54:05.409127 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:05.409137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:05.409170 (+ 33us) server_negotiation.cc:407] Connection header received
0619 00:54:05.409234 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:05.409243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:05.409526 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:05.409695 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:05.410222 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.411825 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.414025 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.414966 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.415125 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:05.416683 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:05.416692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:05.416708 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:05.417419 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:05.417471 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:05.417493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:05.417705 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:05.417930 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 00:54:05.418056 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":388,"threads_started":1}
I20260619 00:54:05.419714 9920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:05.502583 9925 heartbeater.cc:503] Master 127.0.97.62:36763 was elected leader, sending a full tablet report...
I20260619 00:54:05.503731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9798
W20260619 00:54:05.520843 9725 connection.cc:535] server connection from 127.0.97.1:38743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 00:54:05.521204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9929
I20260619 00:54:05.530135 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:05.522168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.97.2:36931)
0619 00:54:05.522717 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:05.522725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:05.522753 (+ 28us) server_negotiation.cc:407] Connection header received
0619 00:54:05.522823 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:05.522830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:05.523074 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:05.523217 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:05.523635 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.524663 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.526230 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.526838 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.527035 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:05.528322 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:05.528332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:05.528352 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:05.529149 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:05.529209 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:05.529232 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:05.529376 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:05.529552 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 00:54:05.529745 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260619 00:54:05.531598 10051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:05.613986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9929
I20260619 00:54:05.631101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10059
I20260619 00:54:05.639940 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:05.632358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.97.3:40033)
0619 00:54:05.633051 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:05.633059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:05.633090 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:54:05.633152 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:05.633158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:05.633416 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:05.633592 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:05.634054 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.635093 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.636389 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:05.637051 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:05.637214 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:05.638389 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:05.638398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:05.638422 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:05.639039 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:05.639089 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:05.639110 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:05.639250 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:05.639412 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 00:54:05.639581 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":338,"threads_started":1}
I20260619 00:54:05.641422 10181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:05.726284 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10059
I20260619 00:54:05.742965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 9708
I20260619 00:54:05.744334 9769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:05.847352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 9708
2026-06-19T00:54:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2588 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T00:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:54:05Z Disabled control of system clock
I20260619 00:54:05.922624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 00:54:06.274392 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:06.274724 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:06.274879 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:06.282416 10205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:54:06.282554 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:06.282629 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:06.282729 10205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:54:06.282804 10205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:54:06.292582 10205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:06.294020 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:06.295964 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:06.309273 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:06.309275 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:06.309293 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:06.310338 10205 server_base.cc:1034] running on GCE node
I20260619 00:54:06.311239 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:06.312924 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:06.314198 10205 hybrid_clock.cc:648] HybridClock initialized: now 1781830446314136 us; error 59 us; skew 500 ppm
I20260619 00:54:06.314639 10205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:06.317279 10205 webserver.cc:460] Webserver started at http://127.0.97.62:45899/ using document root <none> and password file <none>
I20260619 00:54:06.318056 10205 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:06.318198 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:06.318547 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:06.321667 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "5ad2ea54f25544fe8665e7ee12b7738f"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "5b5df16dda28f7b186d16660798bd576"
server_key_iv: "c3c1c71a60984d9647792f80ca08110a"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.322506 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ad2ea54f25544fe8665e7ee12b7738f"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "5b5df16dda28f7b186d16660798bd576"
server_key_iv: "c3c1c71a60984d9647792f80ca08110a"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.328300 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 00:54:06.332290 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:06.334021 10205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:54:06.334236 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "5ad2ea54f25544fe8665e7ee12b7738f"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "5b5df16dda28f7b186d16660798bd576"
server_key_iv: "c3c1c71a60984d9647792f80ca08110a"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.334488 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:06.340855 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:06.341696 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:06.342003 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:06.360833 10205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45049
I20260619 00:54:06.360875 10270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45049 every 8 connection(s)
I20260619 00:54:06.362257 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:54:06.366250 10271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:06.366899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10205
I20260619 00:54:06.367129 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:54:06.367790 388 external_mini_cluster.cc:1428] Setting key 7177db47f002dd9bacfb4c4a53a1ff5c
I20260619 00:54:06.377769 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:06.370055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.97.62:45049)
0619 00:54:06.371323 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:06.371341 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:06.371405 (+ 64us) server_negotiation.cc:407] Connection header received
0619 00:54:06.373341 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:06.373375 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:06.374083 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:06.374613 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:06.375632 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:06.375639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:06.375656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:06.375672 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:06.375750 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:06.375772 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:06.376009 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:06.376137 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 00:54:06.376429 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":389,"threads_started":1}
I20260619 00:54:06.381184 10271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f: Bootstrap starting.
I20260619 00:54:06.384272 10271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:06.385419 10271 log.cc:824] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:06.388597 10271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f: No bootstrap required, opened a new log
I20260619 00:54:06.396314 10271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:54:06.396729 10271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:06.396885 10271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad2ea54f25544fe8665e7ee12b7738f, State: Initialized, Role: FOLLOWER
I20260619 00:54:06.397493 10271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:54:06.397735 10271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:54:06.397879 10271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:54:06.398056 10271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:06.400255 10271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:54:06.400791 10271 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad2ea54f25544fe8665e7ee12b7738f; no voters:
I20260619 00:54:06.401589 10271 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:54:06.401746 10276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:54:06.402462 10276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [term 1 LEADER]: Becoming Leader. State: Replica: 5ad2ea54f25544fe8665e7ee12b7738f, State: Running, Role: LEADER
I20260619 00:54:06.402992 10276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:54:06.403842 10271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:54:06.406281 10278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad2ea54f25544fe8665e7ee12b7738f. Latest consensus state: current_term: 1 leader_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260619 00:54:06.406350 10277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad2ea54f25544fe8665e7ee12b7738f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260619 00:54:06.406746 10278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:06.406744 10277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:06.408466 10284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:54:06.413870 10284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:54:06.423868 10284 catalog_manager.cc:1347] Generated new cluster ID: e3acfd2bc2ec4271a0a0cd6edaf45d34
I20260619 00:54:06.424036 10284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:54:06.447994 10284 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:54:06.448503 10284 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:54:06.461862 10284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad2ea54f25544fe8665e7ee12b7738f: Generated new TSK 0
I20260619 00:54:06.462893 10284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:54:06.467887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 00:54:06.817361 10295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:06.817691 10295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:06.817875 10295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:06.825209 10295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:06.825320 10295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:06.825503 10295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:54:06.835129 10295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:06.836495 10295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:06.838410 10295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:06.852393 10303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:06.852340 10301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:06.852449 10300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:06.853461 10295 server_base.cc:1034] running on GCE node
I20260619 00:54:06.854146 10295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:06.855165 10295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:06.856371 10295 hybrid_clock.cc:648] HybridClock initialized: now 1781830446856311 us; error 53 us; skew 500 ppm
I20260619 00:54:06.856715 10295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:06.859131 10295 webserver.cc:460] Webserver started at http://127.0.97.1:35511/ using document root <none> and password file <none>
I20260619 00:54:06.859866 10295 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:06.859961 10295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:06.860268 10295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:06.864802 10295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "59741fb40522428fb1486ec2a94f3d00"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "b7d6aca76fe25eccf505d65e652b3507"
server_key_iv: "00dddcfa2b77342e5a07aafb265f007d"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.865723 10295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "59741fb40522428fb1486ec2a94f3d00"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "b7d6aca76fe25eccf505d65e652b3507"
server_key_iv: "00dddcfa2b77342e5a07aafb265f007d"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.871181 10295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:54:06.874832 10309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:06.876564 10295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:54:06.876747 10295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "59741fb40522428fb1486ec2a94f3d00"
format_stamp: "Formatted at 2026-06-19 00:54:06 on dist-test-slave-r7bh"
server_key: "b7d6aca76fe25eccf505d65e652b3507"
server_key_iv: "00dddcfa2b77342e5a07aafb265f007d"
server_key_version: "encryptionkey@0"
I20260619 00:54:06.877053 10295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:06.882661 10295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:06.883430 10295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:06.883685 10295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:06.884562 10295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:06.885911 10295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:06.886022 10295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:06.886145 10295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:06.886247 10295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:06.918772 10295 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43591
I20260619 00:54:06.918826 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43591 every 8 connection(s)
I20260619 00:54:06.920171 10295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:54:06.921308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10295
I20260619 00:54:06.921533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:54:06.922194 388 external_mini_cluster.cc:1428] Setting key 9dfc868d45c874e6df2ffc744f011f2d
I20260619 00:54:06.925405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 00:54:06.930955 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:06.923180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54097 (local address 127.0.97.62:45049)
0619 00:54:06.923631 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:06.923637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:06.925327 (+ 1690us) server_negotiation.cc:407] Connection header received
0619 00:54:06.928219 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:06.928225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:06.928283 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:06.928430 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:06.930151 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:06.930157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:06.930160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:06.930181 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:06.930231 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:06.930246 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:06.930400 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:06.930664 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 00:54:06.930723 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260619 00:54:06.932410 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:06.923897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.1:54097)
0619 00:54:06.925146 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:06.925198 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:06.927826 (+ 2628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:06.928747 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:06.928763 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:06.928952 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:06.929713 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:06.929718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:06.929775 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:06.929897 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:06.930387 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:06.930401 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:06.930414 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 00:54:06.930710 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 00:54:06.931028 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":340,"threads_started":1}
I20260619 00:54:06.934201 10422 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:54:06.934594 10422 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:06.935206 10422 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:54:06.937688 10236 ts_manager.cc:194] Registered new tserver with Master: 59741fb40522428fb1486ec2a94f3d00 (127.0.97.1:43591)
W20260619 00:54:07.268909 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:07.269203 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:07.269292 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:07.276654 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:07.276777 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:07.276909 10426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:54:07.286260 10426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:07.287539 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:07.289342 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:07.302495 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:07.302553 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:07.302588 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:07.303786 10426 server_base.cc:1034] running on GCE node
I20260619 00:54:07.304345 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:07.305402 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:07.306608 10426 hybrid_clock.cc:648] HybridClock initialized: now 1781830447306559 us; error 37 us; skew 500 ppm
I20260619 00:54:07.306926 10426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:07.309265 10426 webserver.cc:460] Webserver started at http://127.0.97.2:44231/ using document root <none> and password file <none>
I20260619 00:54:07.309993 10426 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:07.310109 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:07.310424 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:07.313361 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "16dd26c17fca439e8629494bcce36a62"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "f45972a5ce4def8923a7a7ce2d984f2a"
server_key_iv: "9f024cf6e59408398922509409d947ec"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.314172 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16dd26c17fca439e8629494bcce36a62"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "f45972a5ce4def8923a7a7ce2d984f2a"
server_key_iv: "9f024cf6e59408398922509409d947ec"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.319253 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:54:07.322748 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.324064 10426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:54:07.324235 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "16dd26c17fca439e8629494bcce36a62"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "f45972a5ce4def8923a7a7ce2d984f2a"
server_key_iv: "9f024cf6e59408398922509409d947ec"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.324453 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:07.330283 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:07.331190 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:07.331463 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:07.332351 10426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:07.333727 10426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:07.333812 10426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.333973 10426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:07.334046 10426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.367726 10426 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33435
I20260619 00:54:07.367770 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33435 every 8 connection(s)
I20260619 00:54:07.369184 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:54:07.374034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10426
I20260619 00:54:07.374228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:54:07.374621 388 external_mini_cluster.cc:1428] Setting key de73588fe467c5a3098d8de407b26500
I20260619 00:54:07.377816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 00:54:07.378698 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.371943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34177 (local address 127.0.97.62:45049)
0619 00:54:07.372148 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:07.372154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:07.373858 (+ 1704us) server_negotiation.cc:407] Connection header received
0619 00:54:07.376289 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:07.376294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:07.376353 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:07.376468 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:07.377860 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:07.377866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:07.377870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:07.377893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:07.377961 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:07.377978 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:07.378101 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:07.378366 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 00:54:07.378430 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:54:07.379791 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.372526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.2:34177)
0619 00:54:07.373667 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:07.373713 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:07.375940 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:07.376704 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:07.376712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:07.376859 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:07.377544 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:07.377546 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:07.377584 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:07.377669 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:07.378151 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:07.378160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:07.378168 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 00:54:07.378462 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 00:54:07.378731 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":346,"threads_started":1}
I20260619 00:54:07.381278 10553 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:54:07.381599 10553 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:07.382261 10553 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:54:07.383761 10236 ts_manager.cc:194] Registered new tserver with Master: 16dd26c17fca439e8629494bcce36a62 (127.0.97.2:33435)
W20260619 00:54:07.717064 10556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:07.717375 10556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:07.717465 10556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:07.724973 10556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:07.725095 10556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:07.725212 10556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:54:07.735538 10556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:07.736935 10556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:07.738795 10556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:07.753149 10561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:07.753059 10564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:07.753126 10562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:07.754087 10556 server_base.cc:1034] running on GCE node
I20260619 00:54:07.754938 10556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:07.755980 10556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:07.757201 10556 hybrid_clock.cc:648] HybridClock initialized: now 1781830447757145 us; error 53 us; skew 500 ppm
I20260619 00:54:07.757557 10556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:54:07.759979 10556 webserver.cc:460] Webserver started at http://127.0.97.3:38313/ using document root <none> and password file <none>
I20260619 00:54:07.760730 10556 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:07.760923 10556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:07.761251 10556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:07.764210 10556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "070e75ea3d5d43cd811888b2ba66b1e4"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "3f175a899e47e9c493f7ebd6c649e679"
server_key_iv: "15114813677304b68b443c1721148051"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.765144 10556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "070e75ea3d5d43cd811888b2ba66b1e4"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "3f175a899e47e9c493f7ebd6c649e679"
server_key_iv: "15114813677304b68b443c1721148051"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.770716 10556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:54:07.774525 10570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.776181 10556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:54:07.776376 10556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "070e75ea3d5d43cd811888b2ba66b1e4"
format_stamp: "Formatted at 2026-06-19 00:54:07 on dist-test-slave-r7bh"
server_key: "3f175a899e47e9c493f7ebd6c649e679"
server_key_iv: "15114813677304b68b443c1721148051"
server_key_version: "encryptionkey@0"
I20260619 00:54:07.776641 10556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:07.782651 10556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:07.783532 10556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:07.783831 10556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:07.784849 10556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:07.786299 10556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:07.786427 10556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.786561 10556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:07.786648 10556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:07.819748 10556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45261
I20260619 00:54:07.819798 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45261 every 8 connection(s)
I20260619 00:54:07.821168 10556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:54:07.822367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10556
I20260619 00:54:07.822568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:54:07.823104 388 external_mini_cluster.cc:1428] Setting key 153d70a3b46dc3eeb9ddc1fcec63cc53
I20260619 00:54:07.830595 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.824505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43277 (local address 127.0.97.62:45049)
0619 00:54:07.824709 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:07.824715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:07.826661 (+ 1946us) server_negotiation.cc:407] Connection header received
0619 00:54:07.828453 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:07.828457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:07.828519 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:07.828609 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:07.830010 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:07.830015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:07.830018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:07.830035 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:07.830068 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:07.830081 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:07.830184 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:07.830323 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 00:54:07.830373 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:54:07.831640 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.825267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.3:43277)
0619 00:54:07.826484 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:07.826538 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:07.828173 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:07.828798 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:07.828848 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:07.828988 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:07.829697 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:54:07.829699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:07.829739 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:54:07.829827 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:07.830159 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:07.830164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:07.830169 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 00:54:07.830374 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 00:54:07.830614 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":302,"threads_started":1}
I20260619 00:54:07.833225 10683 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:54:07.833540 10683 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:07.834160 10683 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:54:07.835901 10236 ts_manager.cc:194] Registered new tserver with Master: 070e75ea3d5d43cd811888b2ba66b1e4 (127.0.97.3:45261)
I20260619 00:54:07.839135 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 00:54:07.845744 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:54:07.843983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.97.62:45049)
0619 00:54:07.844168 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:07.844173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:07.844681 (+ 508us) server_negotiation.cc:407] Connection header received
0619 00:54:07.844933 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:07.844939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:07.845002 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:07.845113 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:07.845461 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39894: BlockingRecv error: recv got EOF from 127.0.0.1:39894 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:54:07.846362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10295
I20260619 00:54:07.850180 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.847490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.97.1:43591)
0619 00:54:07.848056 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:07.848063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:07.848090 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:54:07.848158 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:07.848164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:07.848384 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:07.848531 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:07.849482 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:07.849489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:07.849502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:07.849521 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:07.849562 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:07.849575 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:07.849686 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:07.849857 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 00:54:07.849943 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":289,"threads_started":1}
I20260619 00:54:07.851451 10417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:07.934633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10295
I20260619 00:54:07.952096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10426
I20260619 00:54:07.956259 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:07.953188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.97.2:33435)
0619 00:54:07.953776 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:07.953784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:07.953822 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:54:07.953885 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:07.953892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:07.954130 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:07.954337 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:07.955466 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:07.955472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:07.955485 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:07.955502 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:07.955543 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:07.955557 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:07.955668 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:07.955919 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 00:54:07.956010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":268,"threads_started":1}
I20260619 00:54:07.957712 10548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:08.042083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10426
I20260619 00:54:08.060058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10556
I20260619 00:54:08.064201 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:08.061088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.0.97.3:45261)
0619 00:54:08.061697 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:08.061704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:08.061733 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:54:08.061803 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:08.061809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:08.062049 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:08.062229 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:08.063428 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:08.063434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:08.063446 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:54:08.063459 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:08.063503 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:54:08.063517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:08.063706 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:08.063783 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 00:54:08.063887 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":305,"threads_started":1}
I20260619 00:54:08.065603 10678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:08.147729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10556
I20260619 00:54:08.164546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10205
I20260619 00:54:08.165939 10266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:08.268386 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10205
2026-06-19T00:54:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2421 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:54:08 dist-test-slave-r7bh krb5kdc[10701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:54:08 dist-test-slave-r7bh krb5kdc[10701](info): set up 2 sockets
Jun 19 00:54:08 dist-test-slave-r7bh krb5kdc[10701](info): commencing operation
krb5kdc: starting...
W20260619 00:54:09.402076 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:54:09 dist-test-slave-r7bh krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:54:09.639685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34027
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:54:09.990067 10714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:09.990393 10714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:09.990602 10714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:09.998009 10714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:54:09.998123 10714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:09.998263 10714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:09.998373 10714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:54:09.998445 10714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:54:10.008446 10714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:10.009867 10714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:10.011204 10714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:10.024171 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:10.024168 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:10.024080 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:10.025008 10714 server_base.cc:1034] running on GCE node
I20260619 00:54:10.025794 10714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:10.027411 10714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:10.028692 10714 hybrid_clock.cc:648] HybridClock initialized: now 1781830450028636 us; error 44 us; skew 500 ppm
I20260619 00:54:10.029139 10714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:54:10 dist-test-slave-r7bh krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:10.032599 10714 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:54:10.034265 10714 webserver.cc:460] Webserver started at http://127.0.97.62:35681/ using document root <none> and password file <none>
I20260619 00:54:10.035001 10714 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:10.035147 10714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:10.035455 10714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:10.038648 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "977c274a79314791b08450cc44828e78"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "cd565415471f4d0e20c8968dad54c1e3"
server_key_iv: "67e41757427a5cb8b55e73555b0c2deb"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.039430 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "977c274a79314791b08450cc44828e78"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "cd565415471f4d0e20c8968dad54c1e3"
server_key_iv: "67e41757427a5cb8b55e73555b0c2deb"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.045066 10714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 00:54:10.048803 10729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:10.050392 10714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:54:10.050612 10714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "977c274a79314791b08450cc44828e78"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "cd565415471f4d0e20c8968dad54c1e3"
server_key_iv: "67e41757427a5cb8b55e73555b0c2deb"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.050830 10714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:10.077733 10714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:10.082301 10714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:10.082646 10714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:10.101320 10714 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34027
I20260619 00:54:10.101332 10780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34027 every 8 connection(s)
I20260619 00:54:10.103024 10714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:54:10.103482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10714
I20260619 00:54:10.103727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:54:10.104573 388 external_mini_cluster.cc:1428] Setting key e77c7e3f6d3567240ae2bca7877eebc9
I20260619 00:54:10.107846 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 00:54:10 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:54:10.121372 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: Bootstrap starting.
I20260619 00:54:10.124549 10781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:10.125691 10781 log.cc:824] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:10.128189 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:10.106746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.97.62:34027)
0619 00:54:10.107687 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:10.107699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:10.107739 (+ 40us) server_negotiation.cc:407] Connection header received
0619 00:54:10.109426 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:10.109449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:10.109842 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:10.110192 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:10.111927 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.113347 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.115120 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.115792 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.115919 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:10.120547 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:10.120557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:10.120577 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:10.120625 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:10.123883 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:10.124599 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:10.124606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:10.124612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:10.124895 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:10.125291 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:10.125297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:10.125301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:10.125969 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:10.126268 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:10.126739 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 00:54:10.127167 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":517,"threads_started":1}
I20260619 00:54:10.128746 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: No bootstrap required, opened a new log
I20260619 00:54:10.136540 10781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } }
I20260619 00:54:10.137007 10781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:10.137122 10781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977c274a79314791b08450cc44828e78, State: Initialized, Role: FOLLOWER
I20260619 00:54:10.137710 10781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } }
I20260619 00:54:10.137955 10781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:54:10.138084 10781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:54:10.138270 10781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:10.140731 10781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } }
I20260619 00:54:10.141367 10781 leader_election.cc:302] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977c274a79314791b08450cc44828e78; no voters:
I20260619 00:54:10.141948 10781 leader_election.cc:288] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:54:10.142078 10786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:54:10.142740 10786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [term 1 LEADER]: Becoming Leader. State: Replica: 977c274a79314791b08450cc44828e78, State: Running, Role: LEADER
I20260619 00:54:10.143241 10786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } }
I20260619 00:54:10.143950 10781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:54:10.146339 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977c274a79314791b08450cc44828e78. Latest consensus state: current_term: 1 leader_uuid: "977c274a79314791b08450cc44828e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } } }
I20260619 00:54:10.146757 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:10.147342 10787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977c274a79314791b08450cc44828e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c274a79314791b08450cc44828e78" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34027 } } }
I20260619 00:54:10.147810 10787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:10.149706 10795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:54:10.154229 10795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:54:10.164038 10795 catalog_manager.cc:1347] Generated new cluster ID: ea1be71009b040f4a0a60c91bb118e19
I20260619 00:54:10.164170 10795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:54:10.186147 10795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:54:10.188031 10795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:54:10.201231 10795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: Generated new TSK 0
I20260619 00:54:10.202376 10795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:54:10.263079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:54:10.617487 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:10.617844 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:10.618022 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:10.625499 10807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:10.625674 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:10.625852 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:54:10.636646 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:10.638166 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:10.639597 10807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:10.654207 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:10.654265 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:10.654162 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:10.660666 10807 server_base.cc:1034] running on GCE node
I20260619 00:54:10.661478 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:10.662599 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:10.663829 10807 hybrid_clock.cc:648] HybridClock initialized: now 1781830450663789 us; error 39 us; skew 500 ppm
I20260619 00:54:10.664243 10807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:54:10 dist-test-slave-r7bh krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:10.667938 10807 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:54:10.669876 10807 webserver.cc:460] Webserver started at http://127.0.97.1:41547/ using document root <none> and password file <none>
I20260619 00:54:10.670763 10807 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:10.670938 10807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:10.671303 10807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:10.674777 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "edcfd3b4c72247dfb129cd636ce94290"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "493c2bde636a9a262d2f8d9c9d0de866"
server_key_iv: "fcef021f9f8b718876c3fd1456e6cc1e"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.675719 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "edcfd3b4c72247dfb129cd636ce94290"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "493c2bde636a9a262d2f8d9c9d0de866"
server_key_iv: "fcef021f9f8b718876c3fd1456e6cc1e"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.681901 10807 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 00:54:10.685865 10822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:10.687363 10807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:54:10.687574 10807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "edcfd3b4c72247dfb129cd636ce94290"
format_stamp: "Formatted at 2026-06-19 00:54:10 on dist-test-slave-r7bh"
server_key: "493c2bde636a9a262d2f8d9c9d0de866"
server_key_iv: "fcef021f9f8b718876c3fd1456e6cc1e"
server_key_version: "encryptionkey@0"
I20260619 00:54:10.687806 10807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:10.715421 10807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:10.720479 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:10.720795 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:10.721874 10807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:10.723292 10807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:10.723404 10807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:10.723554 10807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:10.723626 10807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:10.757970 10807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39219
I20260619 00:54:10.758020 10934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39219 every 8 connection(s)
I20260619 00:54:10.759312 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:54:10.760450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10807
I20260619 00:54:10.760694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:54:10.761512 388 external_mini_cluster.cc:1428] Setting key 631601f44940b00c0705a7b6b727c24c
Jun 19 00:54:10 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:54:10.782612 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:10.762320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47799 (local address 127.0.97.62:34027)
0619 00:54:10.762810 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:10.762816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:10.763978 (+ 1162us) server_negotiation.cc:407] Connection header received
0619 00:54:10.766171 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:10.766180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:10.766281 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:10.766395 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:10.768901 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.770120 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.772388 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.772979 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.773123 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:10.776884 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:10.776890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:10.776902 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:10.776934 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:10.779395 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:10.780174 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:10.780179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:10.780183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:10.780344 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:10.780759 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:10.780766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:10.780771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:10.781083 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:10.781235 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:10.781998 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 00:54:10.782242 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
I20260619 00:54:10.783660 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:10.762958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34027 (local address 127.0.97.1:47799)
0619 00:54:10.763821 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:10.763860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:10.765711 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:10.766878 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:10.766892 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:10.767746 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:10.768645 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:10.768673 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.770337 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.770341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:10.772233 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:10.772247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:10.773285 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:10.773296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:10.773546 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:10.774234 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:10.774257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:10.776639 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:10.779571 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:10.779580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:10.779599 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:10.780046 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:10.780456 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:10.780461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:10.780465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:10.780651 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:10.781224 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:10.781232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:10.781589 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 00:54:10.781889 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 00:54:10.782471 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":390,"threads_started":1}
I20260619 00:54:10.785557 10935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34027
I20260619 00:54:10.785872 10935 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:54:10.786615 10935 heartbeater.cc:511] Master 127.0.97.62:34027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:54:10.789117 10746 ts_manager.cc:194] Registered new tserver with Master: edcfd3b4c72247dfb129cd636ce94290 (127.0.97.1:39219)
I20260619 00:54:10.790655 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:54:10.819784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:54:11.157035 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:11.157328 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:11.157428 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:11.164736 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:11.164933 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:11.165057 10941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:54:11.175243 10941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:11.176548 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:11.177968 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:11.191712 10947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:11.191764 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:11.191679 10946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:11.192780 10941 server_base.cc:1034] running on GCE node
I20260619 00:54:11.193506 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:11.194520 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:11.195758 10941 hybrid_clock.cc:648] HybridClock initialized: now 1781830451195709 us; error 49 us; skew 500 ppm
I20260619 00:54:11.196110 10941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:54:11 dist-test-slave-r7bh krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:11.199370 10941 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:54:11.200894 10941 webserver.cc:460] Webserver started at http://127.0.97.2:46285/ using document root <none> and password file <none>
I20260619 00:54:11.201617 10941 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:11.201713 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:11.201979 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:11.205060 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "0043489de4b64847b2622d47ab679244"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "9e6e894d1261d4d72c6a145ee178870b"
server_key_iv: "6d299cf8c78a682532db14cd9f00c756"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.205812 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0043489de4b64847b2622d47ab679244"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "9e6e894d1261d4d72c6a145ee178870b"
server_key_iv: "6d299cf8c78a682532db14cd9f00c756"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.211498 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 00:54:11.215405 10956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.217216 10941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:54:11.217453 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "0043489de4b64847b2622d47ab679244"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "9e6e894d1261d4d72c6a145ee178870b"
server_key_iv: "6d299cf8c78a682532db14cd9f00c756"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.217651 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:11.242776 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:11.248268 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:11.248633 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:11.249905 10941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:11.251461 10941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:11.251549 10941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.251653 10941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:11.251705 10941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.287294 10941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46359
I20260619 00:54:11.287352 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46359 every 8 connection(s)
I20260619 00:54:11.288733 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:54:11.294224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 10941
I20260619 00:54:11.294435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:54:11.295027 388 external_mini_cluster.cc:1428] Setting key b444a367384bfefd06403e74cb52ad21
Jun 19 00:54:11 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:54:11.313359 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.291536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50571 (local address 127.0.97.62:34027)
0619 00:54:11.292003 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.292008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.293159 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 00:54:11.295735 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.295743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.295850 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.295978 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:11.298240 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.298960 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.301734 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.302176 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.302276 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.307139 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:11.307145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:11.307148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:11.307185 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:11.310077 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.310913 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.310924 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.310927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.311054 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.311520 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.311529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.311531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.311788 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:11.311940 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.312716 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.312977 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260619 00:54:11.314368 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.292065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34027 (local address 127.0.97.2:50571)
0619 00:54:11.292991 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:11.293032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:11.295295 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:11.296324 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:11.296334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:11.297118 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:11.298014 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.298049 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.299129 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.299138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.301555 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.301574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.302426 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.302436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.302823 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.303681 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:11.303718 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:11.306846 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:11.310265 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:11.310276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:11.310295 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:11.310754 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:11.311172 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:11.311177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:11.311181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:11.311389 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:11.311937 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:11.311946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:11.312312 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 00:54:11.312587 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 00:54:11.313183 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":393,"threads_started":1}
I20260619 00:54:11.316107 11069 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34027
I20260619 00:54:11.316637 11069 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:11.317552 11069 heartbeater.cc:511] Master 127.0.97.62:34027 requested a full tablet report, sending...
I20260619 00:54:11.319259 10746 ts_manager.cc:194] Registered new tserver with Master: 0043489de4b64847b2622d47ab679244 (127.0.97.2:46359)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:54:11.320380 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:54:11.349901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:54:11.690061 11075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:11.690442 11075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:11.690614 11075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:11.698413 11075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:11.698593 11075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:11.698788 11075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:54:11.708992 11075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:11.710462 11075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:11.711952 11075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:11.726982 11083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:11.727073 11081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:11.726997 11080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:11.728101 11075 server_base.cc:1034] running on GCE node
I20260619 00:54:11.728770 11075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:11.729841 11075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:11.731053 11075 hybrid_clock.cc:648] HybridClock initialized: now 1781830451731003 us; error 42 us; skew 500 ppm
I20260619 00:54:11.731348 11075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:54:11 dist-test-slave-r7bh krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:11.734687 11075 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:54:11.736343 11075 webserver.cc:460] Webserver started at http://127.0.97.3:33871/ using document root <none> and password file <none>
I20260619 00:54:11.737202 11075 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:11.737357 11075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:11.737668 11075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:54:11.740664 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "c1c14de6b1f24c169573764670b7c9c0"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "525e4db9ed5302fe8de34378e25b3781"
server_key_iv: "00a3fa8c8f004ee386e909e3648b3668"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.741530 11075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c1c14de6b1f24c169573764670b7c9c0"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "525e4db9ed5302fe8de34378e25b3781"
server_key_iv: "00a3fa8c8f004ee386e909e3648b3668"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.747263 11075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:54:11.751222 11090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.752987 11075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:54:11.753191 11075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "c1c14de6b1f24c169573764670b7c9c0"
format_stamp: "Formatted at 2026-06-19 00:54:11 on dist-test-slave-r7bh"
server_key: "525e4db9ed5302fe8de34378e25b3781"
server_key_iv: "00a3fa8c8f004ee386e909e3648b3668"
server_key_version: "encryptionkey@0"
I20260619 00:54:11.753408 11075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:11.768877 11075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:11.773479 11075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:11.773787 11075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:11.774780 11075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:54:11.776156 11075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:54:11.776261 11075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.776410 11075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:54:11.776489 11075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:11.794783 10935 heartbeater.cc:503] Master 127.0.97.62:34027 was elected leader, sending a full tablet report...
I20260619 00:54:11.811551 11075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34187
I20260619 00:54:11.811659 11202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34187 every 8 connection(s)
I20260619 00:54:11.813086 11075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:54:11.814067 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 11075
I20260619 00:54:11.814278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:54:11.814836 388 external_mini_cluster.cc:1428] Setting key 78746793c77928d4a7c96952c8711dab
Jun 19 00:54:11 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:54:11.835953 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.816275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47557 (local address 127.0.97.62:34027)
0619 00:54:11.816780 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.816786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.818283 (+ 1497us) server_negotiation.cc:407] Connection header received
0619 00:54:11.820092 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.820097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.820167 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.820264 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:11.821932 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.822818 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.824449 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.824962 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.825069 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.829088 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:11.829094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:11.829098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:11.829138 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:11.832204 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.833220 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.833225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.833227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.833372 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.833912 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.833921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.833926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.834239 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:11.834412 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.835265 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.835516 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260619 00:54:11.837113 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.817146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34027 (local address 127.0.97.3:47557)
0619 00:54:11.818082 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:11.818150 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:11.819816 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:11.820486 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:11.820495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:11.821141 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:54:11.821753 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.821770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.822961 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.822965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.824283 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.824296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.825207 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.825212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.825450 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.826254 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:54:11.826280 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:54:11.828742 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:54:11.832413 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:11.832425 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:11.832452 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:11.833063 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:11.833503 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:54:11.833509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:54:11.833513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:54:11.833763 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:54:11.834397 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:54:11.834407 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:54:11.834857 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 00:54:11.835155 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 00:54:11.835726 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":309,"threads_started":1}
I20260619 00:54:11.838690 11203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34027
I20260619 00:54:11.839020 11203 heartbeater.cc:464] Registering TS with master...
I20260619 00:54:11.839802 11203 heartbeater.cc:511] Master 127.0.97.62:34027 requested a full tablet report, sending...
I20260619 00:54:11.841482 10746 ts_manager.cc:194] Registered new tserver with Master: c1c14de6b1f24c169573764670b7c9c0 (127.0.97.3:34187)
I20260619 00:54:11.842700 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47557
I20260619 00:54:11.843801 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:54:11.862890 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.849280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.97.62:34027)
0619 00:54:11.849486 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.849491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.849929 (+ 438us) server_negotiation.cc:407] Connection header received
0619 00:54:11.850112 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.850118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.850214 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.850337 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:11.851660 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.852364 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.853904 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.854300 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.854406 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.857347 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:11.857352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:11.857356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:11.857394 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:11.860248 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.861007 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.861016 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.861018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.861148 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:11.861569 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:11.861573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:11.861575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:11.861914 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:11.862097 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.862478 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.862656 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:54:11.867319 10746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:54:11.874336 10746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:54:11.899477 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.887233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46359 (local address 127.0.0.1:52406)
0619 00:54:11.888264 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:11.888319 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:11.888554 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:11.889993 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:11.890002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:11.890041 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:54:11.890519 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.891651 (+ 1132us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.894114 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.894120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.896220 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.896235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.898143 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.898149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.898603 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.898667 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 00:54:11.898813 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 00:54:11.899017 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":186,"spinlock_wait_cycles":4864,"thread_start_us":1037,"threads_started":1}
I20260619 00:54:11.900228 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.887459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.0.97.2:46359)
0619 00:54:11.889246 (+ 1787us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.889258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.889292 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:54:11.889404 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.889414 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.889708 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.891943 (+ 2235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:54:11.892229 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.893932 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.896397 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.897943 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.898066 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.898270 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.899740 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.899915 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612,"thread_start_us":1447,"threads_started":1}
I20260619 00:54:11.901502 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.888577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.97.3:34187)
0619 00:54:11.889571 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.889579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.890094 (+ 515us) server_negotiation.cc:407] Connection header received
0619 00:54:11.890394 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.890403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.890768 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.890958 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:54:11.892292 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.893958 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.898697 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.900158 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.900298 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.900530 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.901008 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.901166 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":1237,"threads_started":1}
I20260619 00:54:11.902285 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.887914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34187 (local address 127.0.0.1:49876)
0619 00:54:11.889993 (+ 2079us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:11.890010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:11.890199 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:11.891510 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:11.891515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:11.891540 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:54:11.891975 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.891988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.894128 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.894131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.898479 (+ 4348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.898492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.900567 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.900574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.900749 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.900781 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:54:11.901758 (+ 977us) client_negotiation.cc:240] Negotiation successful
0619 00:54:11.901889 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"mutex_wait_us":113,"thread_start_us":2118,"threads_started":1}
I20260619 00:54:11.902899 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.889213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39219 (local address 127.0.0.1:50678)
0619 00:54:11.895076 (+ 5863us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:11.895097 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:11.895222 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:11.895902 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:11.895906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:11.895931 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:54:11.896377 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.896385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.897755 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.897760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.900186 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:11.900195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.902116 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.902123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:11.902310 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.902341 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 00:54:11.902433 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 00:54:11.902557 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5689,"mutex_wait_us":147,"thread_start_us":5791,"threads_started":1}
I20260619 00:54:11.903618 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:11.889600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.0.97.1:39219)
0619 00:54:11.894653 (+ 5053us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:11.894662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:11.895258 (+ 596us) server_negotiation.cc:407] Connection header received
0619 00:54:11.895444 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:11.895449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:11.895726 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:11.895919 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:54:11.896549 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.897620 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.900398 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:11.901896 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:11.902039 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:11.902249 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:11.903131 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 00:54:11.903289 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4855,"thread_start_us":4707,"threads_started":1}
I20260619 00:54:11.905850 11138 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262f453a364bdaa9443de62289d16c (DEFAULT_TABLE table=test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:54:11.904337 11004 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262f453a364bdaa9443de62289d16c (DEFAULT_TABLE table=test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:54:11.906641 10870 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262f453a364bdaa9443de62289d16c (DEFAULT_TABLE table=test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:54:11.907207 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262f453a364bdaa9443de62289d16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:11.907675 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262f453a364bdaa9443de62289d16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:11.907935 10870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262f453a364bdaa9443de62289d16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:11.919308 11222 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Bootstrap starting.
I20260619 00:54:11.923210 11224 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Bootstrap starting.
I20260619 00:54:11.924021 11222 tablet_bootstrap.cc:654] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:11.924196 11223 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Bootstrap starting.
I20260619 00:54:11.925437 11222 log.cc:824] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:11.927006 11224 tablet_bootstrap.cc:654] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:11.927392 11223 tablet_bootstrap.cc:654] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:11.928393 11224 log.cc:824] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:11.928458 11223 log.cc:824] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:11.929898 11222 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: No bootstrap required, opened a new log
I20260619 00:54:11.930207 11222 ts_tablet_manager.cc:1397] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 00:54:11.931511 11224 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: No bootstrap required, opened a new log
I20260619 00:54:11.931731 11223 tablet_bootstrap.cc:492] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: No bootstrap required, opened a new log
I20260619 00:54:11.931823 11224 ts_tablet_manager.cc:1397] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 00:54:11.932013 11223 ts_tablet_manager.cc:1397] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 00:54:11.939569 11224 raft_consensus.cc:348] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.940006 11224 raft_consensus.cc:374] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:11.940104 11224 raft_consensus.cc:729] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0043489de4b64847b2622d47ab679244, State: Initialized, Role: FOLLOWER
I20260619 00:54:11.940773 11224 consensus_queue.cc:260] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.941293 11222 raft_consensus.cc:348] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.941766 11222 raft_consensus.cc:374] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:11.941872 11222 raft_consensus.cc:729] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c14de6b1f24c169573764670b7c9c0, State: Initialized, Role: FOLLOWER
I20260619 00:54:11.942121 11069 heartbeater.cc:503] Master 127.0.97.62:34027 was elected leader, sending a full tablet report...
I20260619 00:54:11.942528 11222 consensus_queue.cc:260] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.942699 11224 ts_tablet_manager.cc:1428] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 00:54:11.943722 11223 raft_consensus.cc:348] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.944178 11203 heartbeater.cc:503] Master 127.0.97.62:34027 was elected leader, sending a full tablet report...
I20260619 00:54:11.944212 11223 raft_consensus.cc:374] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:11.944394 11223 raft_consensus.cc:729] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcfd3b4c72247dfb129cd636ce94290, State: Initialized, Role: FOLLOWER
I20260619 00:54:11.944900 11222 ts_tablet_manager.cc:1428] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260619 00:54:11.945250 11223 consensus_queue.cc:260] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:11.947192 11223 ts_tablet_manager.cc:1428] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260619 00:54:12.011860 10936 tablet.cc:2369] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:54:12.041049 11070 tablet.cc:2369] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:54:12.065066 11204 tablet.cc:2369] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:54:12.122444 11229 raft_consensus.cc:482] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:54:12.122730 11229 raft_consensus.cc:504] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:12.124785 11229 leader_election.cc:288] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0043489de4b64847b2622d47ab679244 (127.0.97.2:46359), edcfd3b4c72247dfb129cd636ce94290 (127.0.97.1:39219)
I20260619 00:54:12.130561 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.125208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56607 (local address 127.0.97.2:46359)
0619 00:54:12.125412 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.125419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.125439 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:54:12.125672 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.125680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.125761 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.125845 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:54:12.126420 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.127120 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.128765 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.129634 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.129789 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.129881 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.130159 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.130286 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:54:12.130561 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.125111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46359 (local address 127.0.97.3:56607)
0619 00:54:12.125340 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:12.125362 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:12.125494 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:12.126048 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:12.126053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:12.126088 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:54:12.126252 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:12.126287 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.127265 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.127271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:12.128605 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:12.128617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.129797 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.129802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:12.129993 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.130017 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:54:12.130120 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 00:54:12.130255 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":49}
I20260619 00:54:12.131438 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.125788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44999 (local address 127.0.97.1:39219)
0619 00:54:12.125991 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.125997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.126307 (+ 310us) server_negotiation.cc:407] Connection header received
0619 00:54:12.126482 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.126485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.126543 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.126639 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:54:12.127124 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.127816 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.129567 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.130358 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.130630 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.130736 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.131047 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.131166 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:54:12.131444 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.125495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39219 (local address 127.0.97.3:44999)
0619 00:54:12.126203 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 00:54:12.126217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:54:12.126326 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:54:12.126654 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:54:12.126658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:54:12.126681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:54:12.126983 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:12.126991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.127976 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.127980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:12.129417 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:54:12.129429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.130681 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.130687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:54:12.130868 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.130907 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:54:12.131012 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 00:54:12.131150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":1,"thread_start_us":403,"threads_started":1}
I20260619 00:54:12.131780 11024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262f453a364bdaa9443de62289d16c" candidate_uuid: "c1c14de6b1f24c169573764670b7c9c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0043489de4b64847b2622d47ab679244" is_pre_election: true
I20260619 00:54:12.132372 11024 raft_consensus.cc:2454] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c14de6b1f24c169573764670b7c9c0 in term 0.
I20260619 00:54:12.132720 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262f453a364bdaa9443de62289d16c" candidate_uuid: "c1c14de6b1f24c169573764670b7c9c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edcfd3b4c72247dfb129cd636ce94290" is_pre_election: true
I20260619 00:54:12.133189 11094 leader_election.cc:302] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0043489de4b64847b2622d47ab679244, c1c14de6b1f24c169573764670b7c9c0; no voters:
I20260619 00:54:12.133265 10890 raft_consensus.cc:2454] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c14de6b1f24c169573764670b7c9c0 in term 0.
I20260619 00:54:12.133636 11229 raft_consensus.cc:2781] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:54:12.133782 11229 raft_consensus.cc:482] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:54:12.133914 11229 raft_consensus.cc:3037] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:12.136207 11229 raft_consensus.cc:504] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:12.136937 11229 leader_election.cc:288] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [CANDIDATE]: Term 1 election: Requested vote from peers 0043489de4b64847b2622d47ab679244 (127.0.97.2:46359), edcfd3b4c72247dfb129cd636ce94290 (127.0.97.1:39219)
I20260619 00:54:12.137394 11024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262f453a364bdaa9443de62289d16c" candidate_uuid: "c1c14de6b1f24c169573764670b7c9c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0043489de4b64847b2622d47ab679244"
I20260619 00:54:12.137426 10890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262f453a364bdaa9443de62289d16c" candidate_uuid: "c1c14de6b1f24c169573764670b7c9c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edcfd3b4c72247dfb129cd636ce94290"
I20260619 00:54:12.137615 11024 raft_consensus.cc:3037] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:12.137637 10890 raft_consensus.cc:3037] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:12.139765 10890 raft_consensus.cc:2454] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c14de6b1f24c169573764670b7c9c0 in term 1.
I20260619 00:54:12.139770 11024 raft_consensus.cc:2454] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c14de6b1f24c169573764670b7c9c0 in term 1.
I20260619 00:54:12.140197 11094 leader_election.cc:302] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0043489de4b64847b2622d47ab679244, c1c14de6b1f24c169573764670b7c9c0; no voters:
I20260619 00:54:12.140489 11229 raft_consensus.cc:2781] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:54:12.140995 11229 raft_consensus.cc:686] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 1 LEADER]: Becoming Leader. State: Replica: c1c14de6b1f24c169573764670b7c9c0, State: Running, Role: LEADER
I20260619 00:54:12.141510 11229 consensus_queue.cc:237] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } }
I20260619 00:54:12.146483 10746 catalog_manager.cc:5495] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1c14de6b1f24c169573764670b7c9c0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c1c14de6b1f24c169573764670b7c9c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c14de6b1f24c169573764670b7c9c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34187 } health_report { overall_health: HEALTHY } } }
I20260619 00:54:12.195422 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.189790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.0.97.3:34187)
0619 00:54:12.189981 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.189986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.190005 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:54:12.190134 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.190137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.190209 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.190336 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:54:12.190683 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.191578 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.193211 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.193649 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.193753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.194239 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:54:12.194809 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:54:12.194937 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.195139 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.195212 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:54:12.203878 10890 raft_consensus.cc:1261] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 1 FOLLOWER]: Refusing update from remote peer c1c14de6b1f24c169573764670b7c9c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:54:12.203878 11024 raft_consensus.cc:1261] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 1 FOLLOWER]: Refusing update from remote peer c1c14de6b1f24c169573764670b7c9c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:54:12.204944 11232 consensus_queue.cc:1035] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edcfd3b4c72247dfb129cd636ce94290" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:54:12.205313 11229 consensus_queue.cc:1035] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0043489de4b64847b2622d47ab679244" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:54:12.221484 11237 mvcc.cc:205] Tried to move back new op lower bound from 7298377532216139776 to 7298377531975507968. Current Snapshot: MvccSnapshot[applied={T|T < 7298377532216139776}]
I20260619 00:54:12.232066 10746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:54:12.232378 10746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:54:12.236582 10746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 977c274a79314791b08450cc44828e78: Sending DeleteTablet for 3 replicas of tablet 2c262f453a364bdaa9443de62289d16c
I20260619 00:54:12.238042 10870 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262f453a364bdaa9443de62289d16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:54:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50678
I20260619 00:54:12.237993 11138 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262f453a364bdaa9443de62289d16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:54:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49876
I20260619 00:54:12.238011 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262f453a364bdaa9443de62289d16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:54:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52406
I20260619 00:54:12.240083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10807
I20260619 00:54:12.248898 11246 tablet_replica.cc:330] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: stopping tablet replica
I20260619 00:54:12.249260 11247 tablet_replica.cc:330] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: stopping tablet replica
I20260619 00:54:12.249584 11246 raft_consensus.cc:2229] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:54:12.249846 11247 raft_consensus.cc:2229] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:54:12.250211 11246 raft_consensus.cc:2258] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:54:12.250675 11247 raft_consensus.cc:2258] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:54:12.251479 11245 tablet_replica.cc:330] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: stopping tablet replica
I20260619 00:54:12.253053 11246 ts_tablet_manager.cc:1905] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:54:12.253262 11247 ts_tablet_manager.cc:1905] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:54:12.256130 11245 raft_consensus.cc:2229] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:54:12.258059 11245 raft_consensus.cc:2258] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:54:12.260571 11246 ts_tablet_manager.cc:1918] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:54:12.260751 11246 log.cc:1196] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/2c262f453a364bdaa9443de62289d16c
I20260619 00:54:12.261330 11246 ts_tablet_manager.cc:1939] T 2c262f453a364bdaa9443de62289d16c P edcfd3b4c72247dfb129cd636ce94290: Deleting consensus metadata
I20260619 00:54:12.263329 10733 catalog_manager.cc:4841] TS edcfd3b4c72247dfb129cd636ce94290 (127.0.97.1:39219): tablet 2c262f453a364bdaa9443de62289d16c (table test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]) successfully deleted
I20260619 00:54:12.264421 11245 ts_tablet_manager.cc:1905] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:54:12.265136 11247 ts_tablet_manager.cc:1918] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:54:12.265328 11247 log.cc:1196] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/2c262f453a364bdaa9443de62289d16c
I20260619 00:54:12.265937 11247 ts_tablet_manager.cc:1939] T 2c262f453a364bdaa9443de62289d16c P c1c14de6b1f24c169573764670b7c9c0: Deleting consensus metadata
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:54:12.267915 10731 catalog_manager.cc:4841] TS c1c14de6b1f24c169573764670b7c9c0 (127.0.97.3:34187): tablet 2c262f453a364bdaa9443de62289d16c (table test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]) successfully deleted
I20260619 00:54:12.271693 11245 ts_tablet_manager.cc:1918] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:54:12.271883 11245 log.cc:1196] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/2c262f453a364bdaa9443de62289d16c
I20260619 00:54:12.272449 11245 ts_tablet_manager.cc:1939] T 2c262f453a364bdaa9443de62289d16c P 0043489de4b64847b2622d47ab679244: Deleting consensus metadata
I20260619 00:54:12.274466 10733 catalog_manager.cc:4841] TS 0043489de4b64847b2622d47ab679244 (127.0.97.2:46359): tablet 2c262f453a364bdaa9443de62289d16c (table test-table [id=926fcd9fa7ea4e38b9dbb9328b6b5c48]) successfully deleted
I20260619 00:54:12.275238 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.241551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.0.97.1:39219)
0619 00:54:12.241727 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.241736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.255748 (+ 14012us) server_negotiation.cc:407] Connection header received
0619 00:54:12.256246 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.256253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.256340 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.256421 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:12.258415 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.259640 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.261893 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.262581 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.262685 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.268736 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:12.268751 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:12.268766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:12.268865 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:12.271836 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.272734 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.272745 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.272749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.272966 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.273406 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.273411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.273415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.273792 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:12.274116 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.274570 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.274845 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:54:12.276018 10930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:12.366457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10807
I20260619 00:54:12.384074 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10941
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:54:12.403218 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.385014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.97.2:46359)
0619 00:54:12.385220 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.385227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.385251 (+ 24us) server_negotiation.cc:407] Connection header received
0619 00:54:12.385340 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.385346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.385447 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.385560 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:12.386840 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.388100 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.389549 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.390209 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.390365 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.395440 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:12.395462 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:12.395484 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:12.395548 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:12.399592 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.400578 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.400590 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.400594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.400764 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.401303 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.401308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.401312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.401806 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:12.402137 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.402520 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.402764 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:54:12.404135 11064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:12.499058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10941
I20260619 00:54:12.517323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 11075
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:54:12.534471 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:12.518143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.0.97.3:34187)
0619 00:54:12.518322 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:12.518328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:12.518377 (+ 49us) server_negotiation.cc:407] Connection header received
0619 00:54:12.518600 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:12.518606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:12.518685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:12.518801 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:12.520033 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.521014 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.522458 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:12.522934 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:12.523087 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:12.527578 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:12.527591 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:12.527607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:12.527652 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:12.531159 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.532036 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.532054 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.532057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.532224 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:12.532663 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:12.532669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:12.532672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:12.533155 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:12.533502 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:12.533803 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 00:54:12.534043 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:54:12.535202 11198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:12.657531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 11075
I20260619 00:54:12.679071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 10714
I20260619 00:54:12.680302 10776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:54:12.792125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 10714
2026-06-19T00:54:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4525 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[11255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[11255](info): set up 2 sockets
Jun 19 00:54:12 dist-test-slave-r7bh krb5kdc[11255](info): commencing operation
krb5kdc: starting...
W20260619 00:54:13.989794 388 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:54:14 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:54:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 00:54:14.325361 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6V91bC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/postgres -l logfile start
2026-06-19 00:54:16.862 UTC [11285] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 00:54:16.862 UTC [11285] LOG: listening on IPv4 address "127.0.97.20", port 34167
2026-06-19 00:54:16.865 UTC [11285] LOG: listening on Unix socket "/tmp/.s.PGSQL.34167"
2026-06-19 00:54:16.888 UTC [11288] LOG: database system was shut down at 2026-06-19 00:54:15 UTC
2026-06-19 00:54:16.891 UTC [11285] LOG: database system is ready to accept connections
I20260619 00:54:17.861538 388 mini_postgres.cc:96] Postgres bound to 34167
2026-06-19 00:54:17.879 UTC [11296] FATAL: database "slave" does not exist
127.0.97.20:34167 - accepting connections
I20260619 00:54:17.880208 388 mini_ranger.cc:161] Starting Ranger...
I20260619 00:54:17.907496 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 00:54:18.408993 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 00:54:18.409148 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin
2026-06-19 00:54:18,706 [I] DB FLAVOR :POSTGRES
2026-06-19 00:54:18,706 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 00:54:18,706 [I] Checking connection..
2026-06-19 00:54:18,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 00:54:18,963 [I] Checking connection passed.
2026-06-19 00:54:18,963 [I] --------- Verifying version history table ---------
2026-06-19 00:54:18,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:54:19,202 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 00:54:19,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:54:19,460 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 00:54:19,460 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 00:54:19,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 00:54:19.698 UTC [11360] WARNING: there is no transaction in progress
2026-06-19 00:54:19,705 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 00:54:19,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:54:19,960 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 00:54:19,961 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 00:54:19,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 00:54:20,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 00:54:20,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7bh.c.gcp-upstream.internal','N') ;"
2026-06-19 00:54:20,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 00:54:20,951 [I] Table x_portal_user does not exist in database ranger
2026-06-19 00:54:20,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 00:54:21,222 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 00:54:21,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 00:54:21,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 00:54:21,723 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 00:54:21,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 00:54:22.760 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.779 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.797 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.814 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.834 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.852 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.910 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.917 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.932 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.943 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.953 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.966 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.976 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.984 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:22.993 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:23.001 UTC [11468] WARNING: there is no transaction in progress
2026-06-19 00:54:23,674 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 00:54:23,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 00:54:23,935 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 00:54:23,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 00:54:24,218 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 00:54:24,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 00:54:24,487 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 00:54:24,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 00:54:24,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 00:54:25,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7bh.c.gcp-upstream.internal';"
2026-06-19 00:54:25,261 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 00:54:25,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 00:54:25,525 [I] DB_PATCHES have already been applied
I20260619 00:54:25.531030 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6V91bC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 00:54:25.531160 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 00:54:25.546913 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:43845
Jun 19, 2026 12:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:54:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:54:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 00:54:26.703007 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.823s user 0.006s sys 0.038s
Jun 19, 2026 12:54:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:54:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 12:54:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:54:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:54:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 00:54:49.196951 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 00:54:49 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830489, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 00:54:49.375964 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 00:54:49.400202 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 00:54:50.211057 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 00:54:50.211237 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms
I20260619 00:54:50.211375 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34167
2026-06-19 00:54:50,632 [I] DB FLAVOR :POSTGRES
2026-06-19 00:54:50,632 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 00:54:50,632 [I] Checking connection
2026-06-19 00:54:50,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 00:54:50,913 [I] connection success
2026-06-19 00:54:50,913 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 00:54:50,913 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 00:54:50,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 00:54:51,197 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 00:54:51,197 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 00:54:51,198 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 00:54:51,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 00:54:51,507 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 00:54:52.011826 388 mini_ranger_kms.cc:310] Created kms service
I20260619 00:54:52.203748 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 00:54:52.249644 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 00:54:52 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830489, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:54:52.732733 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 00:54:52.733115 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 00:54:52.733184 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 00:54:52.757534 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11649
I20260619 00:54:52.757717 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:47549
Jun 19, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:54:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:54:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 00:54:55 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:55.077544 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.702s user 0.140s sys 0.299s
I20260619 00:54:55.077817 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 00:54:55.078040 388 mini_ranger_kms.cc:401] 127.0.97.20:47549/kms/v1/keys
Jun 19 00:54:55 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 00:54:55 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 00:54:58 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:54:58.460078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36903
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36903
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47549/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:54:58.833279 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:54:58.833599 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:54:58.834086 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:54:58.841596 11699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:54:58.841715 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:54:58.841760 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:54:58.841820 11699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:54:58.841868 11699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:54:58.852138 11699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:47549/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36903
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:54:58.853497 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:54:58.854802 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:54:58.867754 11707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:58.867811 11704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:54:58.867830 11705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:54:58.868883 11699 server_base.cc:1034] running on GCE node
I20260619 00:54:58.869820 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:54:58.871416 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:54:58.872648 11699 hybrid_clock.cc:648] HybridClock initialized: now 1781830498872588 us; error 59 us; skew 500 ppm
I20260619 00:54:58.873093 11699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:54:58 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:54:58.880432 11699 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:54:58.882328 11699 webserver.cc:460] Webserver started at http://127.0.97.62:46733/ using document root <none> and password file <none>
I20260619 00:54:58.883494 11699 fs_manager.cc:359] Metadata directory not provided
I20260619 00:54:58.883621 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:54:58.883929 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:54:58 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:54:58.942492 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "2495a7a93388472697fc44c35f4e65d9"
format_stamp: "Formatted at 2026-06-19 00:54:58 on dist-test-slave-r7bh"
server_key: "5a36b3b8f75256aa8c65abc877846dd1"
server_key_iv: "b32d1d41170439a6e1a7aeed23f7809a"
server_key_version: "kuduclusterkey@0"
I20260619 00:54:58.943667 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "2495a7a93388472697fc44c35f4e65d9"
format_stamp: "Formatted at 2026-06-19 00:54:58 on dist-test-slave-r7bh"
server_key: "5a36b3b8f75256aa8c65abc877846dd1"
server_key_iv: "b32d1d41170439a6e1a7aeed23f7809a"
server_key_version: "kuduclusterkey@0"
I20260619 00:54:58.951256 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260619 00:54:58.973023 11714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:54:58.975097 11699 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260619 00:54:58.975359 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "2495a7a93388472697fc44c35f4e65d9"
format_stamp: "Formatted at 2026-06-19 00:54:58 on dist-test-slave-r7bh"
server_key: "5a36b3b8f75256aa8c65abc877846dd1"
server_key_iv: "b32d1d41170439a6e1a7aeed23f7809a"
server_key_version: "kuduclusterkey@0"
I20260619 00:54:58.975700 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:54:59.009418 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:54:59.014748 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:54:59.015079 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:54:59.037652 11699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36903
I20260619 00:54:59.037654 11765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36903 every 8 connection(s)
I20260619 00:54:59.039501 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:54:59.040442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 11699
I20260619 00:54:59.040647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:54:59.041592 388 external_mini_cluster.cc:1428] Setting key 701c9992dd787c80a64f81e25dae47fb
I20260619 00:54:59.045356 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:54:59.063337 11766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: Bootstrap starting.
Jun 19 00:54:59 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:54:59.067481 11766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: Neither blocks nor log segments found. Creating new log.
I20260619 00:54:59.069326 11766 log.cc:824] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: Log is configured to *not* fsync() on all Append() calls
I20260619 00:54:59.072781 11766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: No bootstrap required, opened a new log
I20260619 00:54:59.078220 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:54:59.043875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.97.62:36903)
0619 00:54:59.049925 (+ 6050us) server_negotiation.cc:206] Beginning negotiation
0619 00:54:59.049943 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 00:54:59.050572 (+ 629us) server_negotiation.cc:407] Connection header received
0619 00:54:59.054071 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:54:59.054112 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:54:59.054842 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:54:59.055397 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:54:59.057146 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:59.059639 (+ 2493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:59.061599 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:54:59.062770 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:54:59.062994 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:54:59.068461 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:54:59.068482 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:54:59.068507 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:54:59.068569 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:54:59.072699 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:59.073833 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:59.073847 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:59.073857 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:59.074070 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:54:59.074555 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:54:59.074563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:54:59.074567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:54:59.075349 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:54:59.075669 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 00:54:59.076174 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 00:54:59.076656 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5743,"thread_start_us":5481,"threads_started":1}
I20260619 00:54:59.082427 11766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } }
I20260619 00:54:59.082826 11766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:54:59.082927 11766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495a7a93388472697fc44c35f4e65d9, State: Initialized, Role: FOLLOWER
I20260619 00:54:59.083530 11766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } }
I20260619 00:54:59.083796 11766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:54:59.083933 11766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:54:59.084507 11766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:54:59.086771 11766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } }
I20260619 00:54:59.087292 11766 leader_election.cc:302] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2495a7a93388472697fc44c35f4e65d9; no voters:
I20260619 00:54:59.088110 11766 leader_election.cc:288] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:54:59.088316 11771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:54:59.090298 11771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [term 1 LEADER]: Becoming Leader. State: Replica: 2495a7a93388472697fc44c35f4e65d9, State: Running, Role: LEADER
I20260619 00:54:59.090794 11766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:54:59.091107 11771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } }
I20260619 00:54:59.095088 11773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2495a7a93388472697fc44c35f4e65d9. Latest consensus state: current_term: 1 leader_uuid: "2495a7a93388472697fc44c35f4e65d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } } }
I20260619 00:54:59.095594 11773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:59.096318 11772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2495a7a93388472697fc44c35f4e65d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2495a7a93388472697fc44c35f4e65d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36903 } } }
I20260619 00:54:59.096657 11772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9 [sys.catalog]: This master's current role is: LEADER
I20260619 00:54:59.106729 11777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:54:59.117238 11777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:54:59.139892 11777 catalog_manager.cc:1347] Generated new cluster ID: 8db234ff10bc4e858fcb09c741992d11
I20260619 00:54:59.140049 11777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:54:59.152781 11766 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 00:54:59.158560 11777 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:54:59.160653 11777 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:54:59.191243 11777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: Generated new TSK 0
I20260619 00:54:59.192602 11777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:55:00.034617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47549/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:55:00 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 00:55:00.521111 11812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:00.521574 11812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:00.521734 11812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:00.534032 11812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:00.534228 11812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:00.534430 11812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:55:00.550778 11812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:47549/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:00.552656 11812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:00.554626 11812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:00.576861 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:00.577585 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:00.586288 11812 server_base.cc:1034] running on GCE node
W20260619 00:55:00.586118 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:00.587394 11812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:00.588891 11812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:00.590178 11812 hybrid_clock.cc:648] HybridClock initialized: now 1781830500590106 us; error 65 us; skew 500 ppm
I20260619 00:55:00.590608 11812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:00 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:00.595976 11812 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:55:00.598299 11812 webserver.cc:460] Webserver started at http://127.0.97.1:44057/ using document root <none> and password file <none>
I20260619 00:55:00.599442 11812 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:00.599597 11812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:00.599974 11812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:00 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:00.624033 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "249dc6f3d266459398f20cdb08d73ea2"
format_stamp: "Formatted at 2026-06-19 00:55:00 on dist-test-slave-r7bh"
server_key: "e5d5603f01d51e65e7c5eb2d74ae3298"
server_key_iv: "32dd8e48afd619c49422fe2ab387dda5"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:00.625402 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "249dc6f3d266459398f20cdb08d73ea2"
format_stamp: "Formatted at 2026-06-19 00:55:00 on dist-test-slave-r7bh"
server_key: "e5d5603f01d51e65e7c5eb2d74ae3298"
server_key_iv: "32dd8e48afd619c49422fe2ab387dda5"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:00.643285 11812 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.009s sys 0.006s
I20260619 00:55:00.681514 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:00.683517 11812 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.004s
I20260619 00:55:00.683763 11812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "249dc6f3d266459398f20cdb08d73ea2"
format_stamp: "Formatted at 2026-06-19 00:55:00 on dist-test-slave-r7bh"
server_key: "e5d5603f01d51e65e7c5eb2d74ae3298"
server_key_iv: "32dd8e48afd619c49422fe2ab387dda5"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:00.684011 11812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:00.729962 11812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:00.748991 11812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:00.749404 11812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:00.766732 11812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:00.785822 11812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:00.785974 11812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:00.786125 11812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:00.786206 11812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:55:00 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:01.205207 11812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41467
I20260619 00:55:01.205253 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41467 every 8 connection(s)
I20260619 00:55:01.207046 11812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:55:01.207962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 11812
I20260619 00:55:01.208173 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:55:01.209077 388 external_mini_cluster.cc:1428] Setting key cfff4a152bff344fcdefc1075e8418b2
Jun 19 00:55:01 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:01.257360 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:01.215080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58475 (local address 127.0.97.62:36903)
0619 00:55:01.225831 (+ 10751us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:01.225837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:01.225859 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:55:01.225921 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:01.225926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:01.226018 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:01.226126 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:01.231282 (+ 5156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:01.232438 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:01.234927 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:01.235648 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:01.235768 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:01.248805 (+ 13037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:01.248847 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:01.248855 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:01.248949 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:01.253122 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:01.254431 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:01.254437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:01.254441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:01.254613 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:01.255240 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:01.255245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:01.255247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:01.255586 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:01.255713 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:01.256708 (+ 995us) server_negotiation.cc:299] Negotiation successful
0619 00:55:01.256983 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10606,"thread_start_us":10491,"threads_started":1}
I20260619 00:55:01.260022 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:01.213921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36903 (local address 127.0.97.1:58475)
0619 00:55:01.221513 (+ 7592us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:01.221589 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:01.224010 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:01.229246 (+ 5236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:01.229261 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:01.229919 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:01.231021 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:01.231048 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:01.232678 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:01.232683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:01.234726 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:01.234743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:01.236159 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:01.236165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:01.236457 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:01.245271 (+ 8814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:01.245311 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:01.248370 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:01.253399 (+ 5029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:01.253410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:01.253431 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:01.254142 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:01.254774 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:01.254779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:01.254782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:01.255092 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:01.255804 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:01.255812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:01.256278 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 00:55:01.257929 (+ 1651us) client_negotiation.cc:240] Negotiation successful
0619 00:55:01.258527 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7196,"thread_start_us":6932,"threads_started":1}
I20260619 00:55:01.262356 11945 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36903
I20260619 00:55:01.262742 11945 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:01.263679 11945 heartbeater.cc:511] Master 127.0.97.62:36903 requested a full tablet report, sending...
I20260619 00:55:01.266700 11731 ts_manager.cc:194] Registered new tserver with Master: 249dc6f3d266459398f20cdb08d73ea2 (127.0.97.1:41467)
I20260619 00:55:01.268713 11731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58475
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:55:01.314599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47549/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:55:01 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 00:55:01.827925 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:01.828236 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:01.828394 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:01.837172 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:01.837301 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:01.837483 11951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:55:01.847757 11951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:47549/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:01.849272 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:01.850652 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:01.867144 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:01.867081 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:01.868280 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:01.868336 11951 server_base.cc:1034] running on GCE node
I20260619 00:55:01.869614 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:01.871235 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:01.872494 11951 hybrid_clock.cc:648] HybridClock initialized: now 1781830501872442 us; error 49 us; skew 500 ppm
I20260619 00:55:01.872910 11951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:01 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:01.877029 11951 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:55:01.878928 11951 webserver.cc:460] Webserver started at http://127.0.97.2:42015/ using document root <none> and password file <none>
I20260619 00:55:01.879804 11951 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:01.879936 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:01.880288 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:01 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:01.899915 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "c0ee7fcfbc5c4a1e9962352800a94745"
format_stamp: "Formatted at 2026-06-19 00:55:01 on dist-test-slave-r7bh"
server_key: "aaf99d7bb1f0701359492927acc55814"
server_key_iv: "f9b7128ff2c10feb14170ee1888f6266"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:01.901230 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ee7fcfbc5c4a1e9962352800a94745"
format_stamp: "Formatted at 2026-06-19 00:55:01 on dist-test-slave-r7bh"
server_key: "aaf99d7bb1f0701359492927acc55814"
server_key_iv: "f9b7128ff2c10feb14170ee1888f6266"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:01.909442 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 00:55:01.930810 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:01.932619 11951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 00:55:01.932924 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "c0ee7fcfbc5c4a1e9962352800a94745"
format_stamp: "Formatted at 2026-06-19 00:55:01 on dist-test-slave-r7bh"
server_key: "aaf99d7bb1f0701359492927acc55814"
server_key_iv: "f9b7128ff2c10feb14170ee1888f6266"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:01.933189 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:01.976097 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:01.980293 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:01.980614 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:01.981930 11951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:01.983661 11951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:01.983767 11951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:01.983903 11951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:01.983973 11951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:02.028087 11951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35005
I20260619 00:55:02.028111 12087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35005 every 8 connection(s)
I20260619 00:55:02.029692 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:55:02.033056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 11951
I20260619 00:55:02.033236 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:55:02.033797 388 external_mini_cluster.cc:1428] Setting key 80d3b7519bda5a397363030d86ef723e
Jun 19 00:55:02 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:02.061146 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.032696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40803 (local address 127.0.97.62:36903)
0619 00:55:02.041398 (+ 8702us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.041404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.041427 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:55:02.041485 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.041491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.041575 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.043977 (+ 2402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:02.044124 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.045289 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.047639 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.048026 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.048178 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.052261 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:02.052268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:02.052274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:02.052335 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:02.056231 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.057518 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.057525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.057528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.057721 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.058452 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.058459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.058462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.058855 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:02.059015 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.060146 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.060452 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8529,"thread_start_us":8410,"threads_started":1}
I20260619 00:55:02.062203 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.033501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36903 (local address 127.0.97.2:40803)
0619 00:55:02.035619 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.035678 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.038474 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.042132 (+ 3658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.042147 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.042817 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:02.043743 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.043777 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.045490 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.045496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.047444 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.047459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.048313 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.048321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.048583 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.049426 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:02.049451 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:02.051890 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:02.056496 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:02.056511 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:02.056536 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:02.057330 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:02.057900 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:02.057909 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:02.057915 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:02.058235 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:02.059131 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:02.059144 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:02.059654 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.059987 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.060576 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"thread_start_us":1530,"threads_started":1}
I20260619 00:55:02.064683 12088 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36903
I20260619 00:55:02.065084 12088 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:02.066162 12088 heartbeater.cc:511] Master 127.0.97.62:36903 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:55:02.068150 11731 ts_manager.cc:194] Registered new tserver with Master: c0ee7fcfbc5c4a1e9962352800a94745 (127.0.97.2:35005)
I20260619 00:55:02.069125 11731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:55:02.100770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:47549/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:55:02.273514 11945 heartbeater.cc:503] Master 127.0.97.62:36903 was elected leader, sending a full tablet report...
W20260619 00:55:02.476680 12094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:02.477067 12094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:02.477176 12094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:02.484774 12094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:02.484925 12094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:02.485041 12094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:55:02.495807 12094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:47549/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:02.497184 12094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:02.498510 12094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:02.512208 12100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:02.512123 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:02.512149 12099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:02.513366 12094 server_base.cc:1034] running on GCE node
I20260619 00:55:02.514032 12094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:02.515077 12094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:02.516755 12094 hybrid_clock.cc:648] HybridClock initialized: now 1781830502516264 us; error 65 us; skew 500 ppm
I20260619 00:55:02.517210 12094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:02 dist-test-slave-r7bh krb5kdc[11255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:02.520547 12094 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:55:02.522306 12094 webserver.cc:460] Webserver started at http://127.0.97.3:46519/ using document root <none> and password file <none>
I20260619 00:55:02.522993 12094 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:02.523097 12094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:02.523355 12094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:02 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:02.540521 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "f8d0aa0342724463aeb56bbc8d4d710d"
format_stamp: "Formatted at 2026-06-19 00:55:02 on dist-test-slave-r7bh"
server_key: "e29d5ecf01a4fc100e1d89e967be8001"
server_key_iv: "0d298f37bd8f19477b15f223fa0dbfcb"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:02.541756 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f8d0aa0342724463aeb56bbc8d4d710d"
format_stamp: "Formatted at 2026-06-19 00:55:02 on dist-test-slave-r7bh"
server_key: "e29d5ecf01a4fc100e1d89e967be8001"
server_key_iv: "0d298f37bd8f19477b15f223fa0dbfcb"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:02.549131 12094 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 00:55:02.567689 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:02.569633 12094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:55:02.569845 12094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "f8d0aa0342724463aeb56bbc8d4d710d"
format_stamp: "Formatted at 2026-06-19 00:55:02 on dist-test-slave-r7bh"
server_key: "e29d5ecf01a4fc100e1d89e967be8001"
server_key_iv: "0d298f37bd8f19477b15f223fa0dbfcb"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:02.570073 12094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:02.589622 12094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:02.594259 12094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:02.594532 12094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:02.595475 12094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:02.596966 12094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:02.597043 12094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:02.597133 12094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:02.597170 12094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:02.634368 12094 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34033
I20260619 00:55:02.634407 12221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34033 every 8 connection(s)
I20260619 00:55:02.635797 12094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:55:02.637740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 12094
I20260619 00:55:02.637928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:55:02.638528 388 external_mini_cluster.cc:1428] Setting key c8b774e52b8ed63a2437a3c34d94aa2b
Jun 19 00:55:02 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:02.659843 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.639581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59967 (local address 127.0.97.62:36903)
0619 00:55:02.640112 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.640118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.641510 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 00:55:02.643690 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.643696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.643807 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.643935 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:02.645567 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.646367 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.648037 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.648485 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.648649 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.652456 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:02.652465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:02.652470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:02.652518 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:02.655842 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.656718 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.656729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.656731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.657377 (+ 646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.657882 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.657888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.657892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.658220 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:02.658393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.659175 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.659419 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":264,"threads_started":1}
I20260619 00:55:02.660951 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.640366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36903 (local address 127.0.97.3:59967)
0619 00:55:02.641353 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.641394 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.643365 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.644198 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.644207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.644687 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:02.645388 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.645407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.646546 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.646551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.647867 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.647881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.648747 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.648755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.649109 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.649897 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:02.649923 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:02.652139 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:02.656018 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:02.656029 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:02.656049 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:02.656562 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:02.657509 (+ 947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:02.657515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:02.657519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:02.657749 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:02.658375 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:02.658385 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:02.658771 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.659070 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.659659 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":486,"threads_started":1}
I20260619 00:55:02.662549 12222 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36903
I20260619 00:55:02.662834 12222 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:02.663574 12222 heartbeater.cc:511] Master 127.0.97.62:36903 requested a full tablet report, sending...
I20260619 00:55:02.665190 11731 ts_manager.cc:194] Registered new tserver with Master: f8d0aa0342724463aeb56bbc8d4d710d (127.0.97.3:34033)
I20260619 00:55:02.666146 11731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59967
I20260619 00:55:02.667629 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:55:02.693328 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.679505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.0.97.62:36903)
0619 00:55:02.679750 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.679758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.680139 (+ 381us) server_negotiation.cc:407] Connection header received
0619 00:55:02.680340 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.680345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.680422 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.680510 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:02.682024 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.682929 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.684627 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.685177 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.685295 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.687686 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:02.687691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:02.687694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:02.687727 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:02.690480 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.691286 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.691290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.691293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.691428 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:02.691863 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:02.691867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:02.691869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:02.692187 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:02.692384 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.692714 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.692960 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:55:02.698277 11731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:55:02.705580 11731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:55:02.743204 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.720028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.0.97.2:35005)
0619 00:55:02.720564 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.720573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.730501 (+ 9928us) server_negotiation.cc:407] Connection header received
0619 00:55:02.732245 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.732261 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.732651 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.732891 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:02.735212 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.736864 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.740758 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.742090 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.742247 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.742467 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.742693 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.742873 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260619 00:55:02.743289 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.719493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35005 (local address 127.0.0.1:55820)
0619 00:55:02.730311 (+ 10818us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.730362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.731513 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.733349 (+ 1836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.733360 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.733406 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:02.733953 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.733977 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.737348 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.737353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.740589 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.740600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.742294 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.742298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.742450 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.742488 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.742657 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.742881 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10533,"mutex_wait_us":9,"thread_start_us":17473,"threads_started":1}
I20260619 00:55:02.746778 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.721090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34033 (local address 127.0.0.1:58560)
0619 00:55:02.737221 (+ 16131us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.737240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.737381 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.738331 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.738335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.738560 (+ 225us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:02.739523 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.739535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.741557 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.741563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.744305 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.744317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.745876 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.745883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.746051 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.746083 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.746212 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.746350 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15919,"mutex_wait_us":3,"spinlock_wait_cycles":353664,"thread_start_us":16429,"threads_started":1}
I20260619 00:55:02.746778 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.720122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.97.3:34033)
0619 00:55:02.736627 (+ 16505us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.736641 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.737592 (+ 951us) server_negotiation.cc:407] Connection header received
0619 00:55:02.737688 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.737696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.737994 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.738194 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:02.739720 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.741401 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.744468 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.745593 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.745831 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.746073 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.746244 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.746403 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16311,"thread_start_us":16180,"threads_started":1}
I20260619 00:55:02.747741 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 0536df12c6594174b3d5371972055bd4 (DEFAULT_TABLE table=test-table [id=9e408fdf85934296bf52815ea3bc1073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:02.748716 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.721417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41467 (local address 127.0.0.1:57488)
0619 00:55:02.737219 (+ 15802us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.737241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.737483 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.738185 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.738189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.738467 (+ 278us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:02.739533 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.739553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.742541 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.742546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.744798 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.744830 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.747865 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.747872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.748039 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.748070 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.748168 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.748321 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15665,"mutex_wait_us":33,"thread_start_us":8450,"threads_started":1}
I20260619 00:55:02.749007 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.722177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.1:41467)
0619 00:55:02.735926 (+ 13749us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.735936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.737361 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 00:55:02.737692 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.737698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.737995 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.738194 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:02.739856 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.741508 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.744971 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.746356 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.746490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.746695 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.748246 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.748502 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13538,"thread_start_us":13757,"threads_started":1}
I20260619 00:55:02.749334 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0536df12c6594174b3d5371972055bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:02.750643 12157 tablet_service.cc:1467] Processing CreateTablet for tablet 0536df12c6594174b3d5371972055bd4 (DEFAULT_TABLE table=test-table [id=9e408fdf85934296bf52815ea3bc1073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:02.751875 12157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0536df12c6594174b3d5371972055bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:02.753262 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 0536df12c6594174b3d5371972055bd4 (DEFAULT_TABLE table=test-table [id=9e408fdf85934296bf52815ea3bc1073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:02.754722 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0536df12c6594174b3d5371972055bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:02.763564 12241 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Bootstrap starting.
I20260619 00:55:02.767475 12241 tablet_bootstrap.cc:654] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:02.769019 12241 log.cc:824] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:02.771893 12242 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Bootstrap starting.
I20260619 00:55:02.777632 12241 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: No bootstrap required, opened a new log
I20260619 00:55:02.777982 12241 ts_tablet_manager.cc:1397] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20260619 00:55:02.778029 12244 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Bootstrap starting.
I20260619 00:55:02.780966 12244 tablet_bootstrap.cc:654] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:02.781951 12242 tablet_bootstrap.cc:654] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:02.782939 12244 log.cc:824] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:02.783483 12242 log.cc:824] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:02.785871 12244 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: No bootstrap required, opened a new log
I20260619 00:55:02.786140 12244 ts_tablet_manager.cc:1397] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260619 00:55:02.786998 12241 raft_consensus.cc:348] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.787395 12241 raft_consensus.cc:374] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:02.787525 12241 raft_consensus.cc:729] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d0aa0342724463aeb56bbc8d4d710d, State: Initialized, Role: FOLLOWER
I20260619 00:55:02.788528 12242 tablet_bootstrap.cc:492] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: No bootstrap required, opened a new log
I20260619 00:55:02.788362 12241 consensus_queue.cc:260] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.788911 12242 ts_tablet_manager.cc:1397] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260619 00:55:02.790338 12222 heartbeater.cc:503] Master 127.0.97.62:36903 was elected leader, sending a full tablet report...
I20260619 00:55:02.791038 12241 ts_tablet_manager.cc:1428] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260619 00:55:02.797881 12244 raft_consensus.cc:348] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.798431 12244 raft_consensus.cc:374] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:02.798540 12244 raft_consensus.cc:729] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249dc6f3d266459398f20cdb08d73ea2, State: Initialized, Role: FOLLOWER
I20260619 00:55:02.799291 12244 consensus_queue.cc:260] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.800117 12242 raft_consensus.cc:348] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.800570 12242 raft_consensus.cc:374] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:02.800674 12242 raft_consensus.cc:729] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ee7fcfbc5c4a1e9962352800a94745, State: Initialized, Role: FOLLOWER
I20260619 00:55:02.801286 12244 ts_tablet_manager.cc:1428] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260619 00:55:02.801352 12242 consensus_queue.cc:260] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.802779 12088 heartbeater.cc:503] Master 127.0.97.62:36903 was elected leader, sending a full tablet report...
I20260619 00:55:02.803424 12242 ts_tablet_manager.cc:1428] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 00:55:02.888370 12223 tablet.cc:2369] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:55:02.943252 12249 raft_consensus.cc:482] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:55:02.943676 12249 raft_consensus.cc:504] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.945698 12249 leader_election.cc:288] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8d0aa0342724463aeb56bbc8d4d710d (127.0.97.3:34033), 249dc6f3d266459398f20cdb08d73ea2 (127.0.97.1:41467)
I20260619 00:55:02.953466 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.946175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39585 (local address 127.0.97.3:34033)
0619 00:55:02.946380 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.946388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.947339 (+ 951us) server_negotiation.cc:407] Connection header received
0619 00:55:02.947582 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.947586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.947647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.947746 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:02.948421 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.949237 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.951620 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.952411 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.952570 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.952682 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.953053 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.953187 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:55:02.953610 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.946229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59815 (local address 127.0.97.1:41467)
0619 00:55:02.946505 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:02.946514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:02.947427 (+ 913us) server_negotiation.cc:407] Connection header received
0619 00:55:02.947615 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:02.947622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:02.947698 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:02.947801 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:02.948551 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.949385 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.951698 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.952539 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.952723 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.952871 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:02.953186 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 00:55:02.953324 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 00:55:02.953651 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.946166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34033 (local address 127.0.97.2:39585)
0619 00:55:02.947210 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.947227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.947418 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.947765 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.947769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.947792 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:02.948263 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.948273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.949376 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.949383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.951453 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.951465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.952639 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.952645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.952873 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.952913 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.953024 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.953169 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":84,"thread_start_us":539,"threads_started":1}
I20260619 00:55:02.953830 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:02.946193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41467 (local address 127.0.97.2:59815)
0619 00:55:02.947315 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:02.947331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:02.947454 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:02.947828 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:02.947833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:02.947865 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:02.948377 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.948387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.949536 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.949553 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.951509 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:02.951521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:02.952790 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:02.952796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:02.953009 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:02.953035 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 00:55:02.953154 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 00:55:02.953294 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":25,"thread_start_us":651,"threads_started":1}
I20260619 00:55:02.955147 12177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0536df12c6594174b3d5371972055bd4" candidate_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" is_pre_election: true
I20260619 00:55:02.955171 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0536df12c6594174b3d5371972055bd4" candidate_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249dc6f3d266459398f20cdb08d73ea2" is_pre_election: true
I20260619 00:55:02.955790 12177 raft_consensus.cc:2454] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ee7fcfbc5c4a1e9962352800a94745 in term 0.
I20260619 00:55:02.955790 11900 raft_consensus.cc:2454] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ee7fcfbc5c4a1e9962352800a94745 in term 0.
I20260619 00:55:02.956601 11979 leader_election.cc:302] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 249dc6f3d266459398f20cdb08d73ea2, c0ee7fcfbc5c4a1e9962352800a94745, f8d0aa0342724463aeb56bbc8d4d710d; no voters:
I20260619 00:55:02.957120 12249 raft_consensus.cc:2781] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:55:02.957291 12249 raft_consensus.cc:482] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:55:02.957394 12249 raft_consensus.cc:3037] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:02.959535 12249 raft_consensus.cc:504] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.960148 12249 leader_election.cc:288] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [CANDIDATE]: Term 1 election: Requested vote from peers f8d0aa0342724463aeb56bbc8d4d710d (127.0.97.3:34033), 249dc6f3d266459398f20cdb08d73ea2 (127.0.97.1:41467)
I20260619 00:55:02.960670 12177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0536df12c6594174b3d5371972055bd4" candidate_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d0aa0342724463aeb56bbc8d4d710d"
I20260619 00:55:02.960713 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0536df12c6594174b3d5371972055bd4" candidate_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249dc6f3d266459398f20cdb08d73ea2"
I20260619 00:55:02.960958 12177 raft_consensus.cc:3037] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:02.960951 11900 raft_consensus.cc:3037] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:02.964282 12177 raft_consensus.cc:2454] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ee7fcfbc5c4a1e9962352800a94745 in term 1.
I20260619 00:55:02.964282 11900 raft_consensus.cc:2454] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ee7fcfbc5c4a1e9962352800a94745 in term 1.
I20260619 00:55:02.964789 11976 leader_election.cc:302] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ee7fcfbc5c4a1e9962352800a94745, f8d0aa0342724463aeb56bbc8d4d710d; no voters:
I20260619 00:55:02.965241 12249 raft_consensus.cc:2781] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:55:02.966374 12249 raft_consensus.cc:686] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 1 LEADER]: Becoming Leader. State: Replica: c0ee7fcfbc5c4a1e9962352800a94745, State: Running, Role: LEADER
I20260619 00:55:02.966969 12249 consensus_queue.cc:237] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } }
I20260619 00:55:02.971935 11729 catalog_manager.cc:5495] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ee7fcfbc5c4a1e9962352800a94745 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ee7fcfbc5c4a1e9962352800a94745" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 } health_report { overall_health: UNKNOWN } } }
W20260619 00:55:02.977211 11946 tablet.cc:2369] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:55:03.032158 12089 tablet.cc:2369] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:55:03.034916 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:03.028001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.97.2:35005)
0619 00:55:03.028210 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:03.028217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:03.028239 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:55:03.028353 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:03.028357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:03.028457 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:03.028615 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:55:03.029069 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.030152 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.031588 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.032308 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.032427 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:03.033173 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:55:03.034282 (+ 1109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:55:03.034423 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:03.034504 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 00:55:03.034598 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:55:03.044948 11900 raft_consensus.cc:1261] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 1 FOLLOWER]: Refusing update from remote peer c0ee7fcfbc5c4a1e9962352800a94745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:55:03.045136 12177 raft_consensus.cc:1261] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 1 FOLLOWER]: Refusing update from remote peer c0ee7fcfbc5c4a1e9962352800a94745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:55:03.046022 12249 consensus_queue.cc:1035] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "249dc6f3d266459398f20cdb08d73ea2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:55:03.046319 12252 consensus_queue.cc:1035] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d0aa0342724463aeb56bbc8d4d710d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:55:03.059098 12255 mvcc.cc:205] Tried to move back new op lower bound from 7298377740461322240 to 7298377740156948480. Current Snapshot: MvccSnapshot[applied={T|T < 7298377740461322240}]
I20260619 00:55:03.083477 11731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:55:03.083761 11731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:55:03.087198 11731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2495a7a93388472697fc44c35f4e65d9: Sending DeleteTablet for 3 replicas of tablet 0536df12c6594174b3d5371972055bd4
I20260619 00:55:03.088424 12157 tablet_service.cc:1514] Processing DeleteTablet for tablet 0536df12c6594174b3d5371972055bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58560
I20260619 00:55:03.088786 11880 tablet_service.cc:1514] Processing DeleteTablet for tablet 0536df12c6594174b3d5371972055bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57488
I20260619 00:55:03.088837 12023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0536df12c6594174b3d5371972055bd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55820
I20260619 00:55:03.091586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 11812
I20260619 00:55:03.098371 12266 tablet_replica.cc:330] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: stopping tablet replica
I20260619 00:55:03.098857 12266 raft_consensus.cc:2229] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:55:03.099570 12266 raft_consensus.cc:2258] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:55:03.101176 12267 tablet_replica.cc:330] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: stopping tablet replica
I20260619 00:55:03.101591 12267 raft_consensus.cc:2229] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:55:03.102094 12267 raft_consensus.cc:2258] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:55:03.103886 12266 ts_tablet_manager.cc:1905] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:55:03.108062 12267 ts_tablet_manager.cc:1905] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:55:03.111840 12265 tablet_replica.cc:330] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: stopping tablet replica
I20260619 00:55:03.112181 12266 ts_tablet_manager.cc:1918] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:55:03.112406 12266 log.cc:1196] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/0536df12c6594174b3d5371972055bd4
Jun 19 00:55:03 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:55:03.112447 12265 raft_consensus.cc:2229] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:55:03.112990 12265 raft_consensus.cc:2258] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:55:03.113243 12266 ts_tablet_manager.cc:1939] T 0536df12c6594174b3d5371972055bd4 P c0ee7fcfbc5c4a1e9962352800a94745: Deleting consensus metadata
I20260619 00:55:03.115289 11717 catalog_manager.cc:4841] TS c0ee7fcfbc5c4a1e9962352800a94745 (127.0.97.2:35005): tablet 0536df12c6594174b3d5371972055bd4 (table test-table [id=9e408fdf85934296bf52815ea3bc1073]) successfully deleted
I20260619 00:55:03.115515 12265 ts_tablet_manager.cc:1905] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:55:03.116017 12267 ts_tablet_manager.cc:1918] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:55:03.116246 12267 log.cc:1196] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/0536df12c6594174b3d5371972055bd4
I20260619 00:55:03.116783 12267 ts_tablet_manager.cc:1939] T 0536df12c6594174b3d5371972055bd4 P 249dc6f3d266459398f20cdb08d73ea2: Deleting consensus metadata
I20260619 00:55:03.118937 11718 catalog_manager.cc:4841] TS 249dc6f3d266459398f20cdb08d73ea2 (127.0.97.1:41467): tablet 0536df12c6594174b3d5371972055bd4 (table test-table [id=9e408fdf85934296bf52815ea3bc1073]) successfully deleted
I20260619 00:55:03.121845 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:03.093273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.0.97.1:41467)
0619 00:55:03.093510 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:03.093519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:03.100364 (+ 6845us) server_negotiation.cc:407] Connection header received
0619 00:55:03.100558 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:03.100566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:03.100721 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:03.100890 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:03.103593 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.105169 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.107461 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.108040 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.108153 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:03.113887 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:03.113909 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:03.113928 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:03.113971 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:03.118700 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.119624 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.119635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.119638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.119810 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.120178 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.120182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.120185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.120629 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:03.120961 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:03.121220 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 00:55:03.121475 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:55:03.122459 12265 ts_tablet_manager.cc:1918] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:55:03.122665 12265 log.cc:1196] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/0536df12c6594174b3d5371972055bd4
I20260619 00:55:03.122681 11940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:03.123294 12265 ts_tablet_manager.cc:1939] T 0536df12c6594174b3d5371972055bd4 P f8d0aa0342724463aeb56bbc8d4d710d: Deleting consensus metadata
I20260619 00:55:03.126302 11715 catalog_manager.cc:4841] TS f8d0aa0342724463aeb56bbc8d4d710d (127.0.97.3:34033): tablet 0536df12c6594174b3d5371972055bd4 (table test-table [id=9e408fdf85934296bf52815ea3bc1073]) successfully deleted
I20260619 00:55:03.250094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 11812
I20260619 00:55:03.267685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 11951
Jun 19 00:55:03 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:55:03.286427 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:03.268648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.0.97.2:35005)
0619 00:55:03.268863 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:03.268869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:03.268895 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:55:03.268979 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:03.268983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:03.269043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:03.269134 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:03.270783 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.271627 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.273878 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.274403 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.274517 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:03.279860 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:03.279877 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:03.279893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:03.279935 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:03.283465 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.284195 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.284203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.284205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.284367 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.284763 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.284774 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.284777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.285162 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:03.285500 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:03.285819 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 00:55:03.286037 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 00:55:03.287122 12083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:03.400768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 11951
I20260619 00:55:03.419405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 12094
Jun 19 00:55:03 dist-test-slave-r7bh krb5kdc[11255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:55:03.435101 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:03.420125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.0.97.3:34033)
0619 00:55:03.420315 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:03.420319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:03.420338 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:55:03.420468 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:03.420472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:03.420534 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:03.420627 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:03.422032 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.422804 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.424113 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:03.424496 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:03.424595 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:03.429068 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:03.429083 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:03.429100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:03.429135 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:03.432015 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.432893 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.432907 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.432913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.433060 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:03.433497 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:03.433502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:03.433504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:03.433939 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:03.434225 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:03.434504 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 00:55:03.434746 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:55:03.435798 12217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:03.525745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 12094
I20260619 00:55:03.542798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 11699
I20260619 00:55:03.544108 11761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:03.657925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 11699
I20260619 00:55:03.684528 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 00:55:04.020247 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 00:55:04.020463 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 00:55:04.366461 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 00:55:04.366 UTC [11285] LOG: received smart shutdown request
2026-06-19 00:55:04.369 UTC [11285] LOG: background worker "logical replication launcher" (PID 11294) exited with exit code 1
2026-06-19 00:55:04.370 UTC [11289] LOG: shutting down
2026-06-19 00:55:04.535 UTC [11285] LOG: database system is shut down
2026-06-19T00:55:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51863 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:55:04 dist-test-slave-r7bh krb5kdc[12281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:55:04 dist-test-slave-r7bh krb5kdc[12281](info): set up 2 sockets
Jun 19 00:55:04 dist-test-slave-r7bh krb5kdc[12281](info): commencing operation
krb5kdc: starting...
W20260619 00:55:05.765209 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:55:05 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:55:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 00:55:06.101176 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6V91bC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/postgres -l logfile start
2026-06-19 00:55:08.701 UTC [12311] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 00:55:08.701 UTC [12311] LOG: listening on IPv4 address "127.0.97.20", port 44545
2026-06-19 00:55:08.705 UTC [12311] LOG: listening on Unix socket "/tmp/.s.PGSQL.44545"
2026-06-19 00:55:08.727 UTC [12314] LOG: database system was shut down at 2026-06-19 00:55:07 UTC
2026-06-19 00:55:08.731 UTC [12311] LOG: database system is ready to accept connections
I20260619 00:55:09.705673 388 mini_postgres.cc:96] Postgres bound to 44545
2026-06-19 00:55:09.723 UTC [12322] FATAL: database "slave" does not exist
127.0.97.20:44545 - accepting connections
I20260619 00:55:09.723748 388 mini_ranger.cc:161] Starting Ranger...
I20260619 00:55:09.751246 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 00:55:10.265542 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 00:55:10.265669 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin
2026-06-19 00:55:10,550 [I] DB FLAVOR :POSTGRES
2026-06-19 00:55:10,551 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 00:55:10,551 [I] Checking connection..
2026-06-19 00:55:10,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 00:55:10,808 [I] Checking connection passed.
2026-06-19 00:55:10,808 [I] --------- Verifying version history table ---------
2026-06-19 00:55:10,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:55:11,047 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 00:55:11,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:55:11,311 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 00:55:11,311 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 00:55:11,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 00:55:11.544 UTC [12386] WARNING: there is no transaction in progress
2026-06-19 00:55:11,551 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 00:55:11,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 00:55:11,809 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 00:55:11,809 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 00:55:11,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 00:55:12,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 00:55:12,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7bh.c.gcp-upstream.internal','N') ;"
2026-06-19 00:55:12,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 00:55:12,833 [I] Table x_portal_user does not exist in database ranger
2026-06-19 00:55:12,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 00:55:13,101 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 00:55:13,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 00:55:13,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 00:55:13,631 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 00:55:13,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 00:55:14.659 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.676 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.694 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.711 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.729 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.748 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.810 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.818 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.835 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.848 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.859 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.873 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.884 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.891 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.898 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:14.906 UTC [12494] WARNING: there is no transaction in progress
2026-06-19 00:55:15,597 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 00:55:15,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 00:55:15,859 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 00:55:15,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 00:55:16,102 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 00:55:16,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 00:55:16,367 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 00:55:16,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 00:55:16,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 00:55:16,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7bh.c.gcp-upstream.internal';"
2026-06-19 00:55:17,121 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 00:55:17,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 00:55:17,365 [I] DB_PATCHES have already been applied
I20260619 00:55:17.371616 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6V91bC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 00:55:17.371719 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 00:55:17.386353 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:56893
Jun 19, 2026 12:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:55:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:55:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 00:55:18.539042 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.815s user 0.001s sys 0.039s
Jun 19, 2026 12:55:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:55:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 12:55:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:55:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:55:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 00:55:40.416352 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 00:55:40 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830540, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 00:55:40.601207 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 00:55:40.624447 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 00:55:41.535452 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 00:55:41.535573 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms
I20260619 00:55:41.535617 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:44545
2026-06-19 00:55:41,914 [I] DB FLAVOR :POSTGRES
2026-06-19 00:55:41,915 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 00:55:41,915 [I] Checking connection
2026-06-19 00:55:41,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 00:55:42,191 [I] connection success
2026-06-19 00:55:42,192 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 00:55:42,192 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 00:55:42,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 00:55:42,452 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 00:55:42,453 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 00:55:42,453 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 00:55:42,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 00:55:42,730 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 00:55:43.262853 388 mini_ranger_kms.cc:310] Created kms service
I20260619 00:55:43.481138 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 00:55:43.532177 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 00:55:43 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830540, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:43.960446 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 00:55:43.960796 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task6V91bC/build/asan/bin/postgresql.jar:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6V91bC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 00:55:43.960876 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 00:55:43.979014 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12674
I20260619 00:55:43.979161 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:46485
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:55:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 00:55:46.356973 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.756s user 0.089s sys 0.314s
I20260619 00:55:46.357244 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 00:55:46.357414 388 mini_ranger_kms.cc:401] 127.0.97.20:46485/kms/v1/keys
Jun 19 00:55:46 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830546, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 00:55:47 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830546, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 00:55:47 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830546, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 00:55:49 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:55:49.756942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40225
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:55:50.112167 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:50.112538 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:50.112695 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:50.120285 12725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:55:50.120469 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:50.120577 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:50.120682 12725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:55:50.120774 12725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:55:50.131810 12725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:46485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40225
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:50.133375 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:50.134814 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:50.147980 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:50.148491 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:50.148146 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:50.149406 12725 server_base.cc:1034] running on GCE node
I20260619 00:55:50.150403 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:50.152333 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:50.153594 12725 hybrid_clock.cc:648] HybridClock initialized: now 1781830550153521 us; error 60 us; skew 500 ppm
I20260619 00:55:50.154017 12725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:50 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:50.163396 12725 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:55:50.165284 12725 webserver.cc:460] Webserver started at http://127.0.97.62:37495/ using document root <none> and password file <none>
I20260619 00:55:50.166020 12725 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:50.166167 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:50.166481 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:50 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:50.228127 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "95e23bd94c7c4f398fe73b6f9a645c76"
format_stamp: "Formatted at 2026-06-19 00:55:50 on dist-test-slave-r7bh"
server_key: "842ff402213a2da4682329f524f3dc3e"
server_key_iv: "2d6f9ab6b56afed30947c7e6a21819f8"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:50.229143 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "95e23bd94c7c4f398fe73b6f9a645c76"
format_stamp: "Formatted at 2026-06-19 00:55:50 on dist-test-slave-r7bh"
server_key: "842ff402213a2da4682329f524f3dc3e"
server_key_iv: "2d6f9ab6b56afed30947c7e6a21819f8"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:50.235882 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 00:55:50.254196 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:50.255899 12725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 00:55:50.256109 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "95e23bd94c7c4f398fe73b6f9a645c76"
format_stamp: "Formatted at 2026-06-19 00:55:50 on dist-test-slave-r7bh"
server_key: "842ff402213a2da4682329f524f3dc3e"
server_key_iv: "2d6f9ab6b56afed30947c7e6a21819f8"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:50.256341 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:50.273101 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:50.277624 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:50.277966 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:50.298683 12725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40225
I20260619 00:55:50.298702 12791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40225 every 8 connection(s)
I20260619 00:55:50.300172 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:55:50.305336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 12725
I20260619 00:55:50.305579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:55:50.305632 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:50.306263 388 external_mini_cluster.cc:1428] Setting key ae05de280b10078e420903df0ed9f614
I20260619 00:55:50.321710 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Bootstrap starting.
I20260619 00:55:50.326455 12792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:50.328034 12792 log.cc:824] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:50.331178 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: No bootstrap required, opened a new log
Jun 19 00:55:50 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:50.340461 12792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:50.340936 12792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:50.341184 12792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e23bd94c7c4f398fe73b6f9a645c76, State: Initialized, Role: FOLLOWER
I20260619 00:55:50.342063 12792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:50.342337 12792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:55:50.342514 12792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:55:50.342715 12792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:50.345690 12792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:50.346450 12792 leader_election.cc:302] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e23bd94c7c4f398fe73b6f9a645c76; no voters:
I20260619 00:55:50.347193 12792 leader_election.cc:288] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:55:50.347318 12797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:55:50.347885 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:50.308576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.0.97.62:40225)
0619 00:55:50.323111 (+ 14535us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:50.323132 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:50.323192 (+ 60us) server_negotiation.cc:407] Connection header received
0619 00:55:50.325138 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:50.325163 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:50.325654 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:50.326101 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:50.327523 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:50.329269 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:50.331169 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:50.332249 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:50.332429 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:50.337662 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:50.337675 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:50.337707 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:50.337771 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:50.342175 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:50.343206 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:50.343216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:50.343226 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:50.343462 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:50.343997 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:50.344003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:50.344006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:50.344948 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:50.345313 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:50.345803 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 00:55:50.346309 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14228,"thread_start_us":14371,"threads_started":1}
I20260619 00:55:50.353605 12797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 LEADER]: Becoming Leader. State: Replica: 95e23bd94c7c4f398fe73b6f9a645c76, State: Running, Role: LEADER
I20260619 00:55:50.354342 12797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:50.354974 12792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:55:50.357573 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } } }
I20260619 00:55:50.358111 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: This master's current role is: LEADER
I20260619 00:55:50.358834 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e23bd94c7c4f398fe73b6f9a645c76. Latest consensus state: current_term: 1 leader_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } } }
I20260619 00:55:50.359220 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: This master's current role is: LEADER
I20260619 00:55:50.365589 12803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:55:50.372743 12803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:55:50.397397 12803 catalog_manager.cc:1347] Generated new cluster ID: 526f4c37eef14eaa9fb6927e8868d5f6
I20260619 00:55:50.397569 12803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:55:50.424780 12803 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:55:50.427441 12803 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:55:50.447499 12803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Generated new TSK 0
I20260619 00:55:50.449365 12803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:55:50.450652 12792 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:55:51.396970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:55:51 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 00:55:51.853518 12837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:51.854079 12837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:51.854291 12837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:51.866739 12837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:51.866997 12837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:51.867265 12837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:55:51.883875 12837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:51.886088 12837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:51.887795 12837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:51.906077 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:51.909286 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:51.920048 12837 server_base.cc:1034] running on GCE node
W20260619 00:55:51.921123 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:51.922173 12837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:51.932621 12837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:51.933990 12837 hybrid_clock.cc:648] HybridClock initialized: now 1781830551933872 us; error 147 us; skew 500 ppm
I20260619 00:55:51.934566 12837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:51 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:51.939473 12837 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:55:51.941960 12837 webserver.cc:460] Webserver started at http://127.0.97.1:45249/ using document root <none> and password file <none>
I20260619 00:55:51.943089 12837 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:51.943302 12837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:51.943768 12837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:51 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:51.964532 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "5c86629c66604aceb8d0895b979322ce"
format_stamp: "Formatted at 2026-06-19 00:55:51 on dist-test-slave-r7bh"
server_key: "8c8bfb18527d836abd20b7100e6110c1"
server_key_iv: "83959268293fe73cd012a875cccec0f9"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:51.965751 12837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5c86629c66604aceb8d0895b979322ce"
format_stamp: "Formatted at 2026-06-19 00:55:51 on dist-test-slave-r7bh"
server_key: "8c8bfb18527d836abd20b7100e6110c1"
server_key_iv: "83959268293fe73cd012a875cccec0f9"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:51.973794 12837 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260619 00:55:52.008461 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:52.011114 12837 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.005s
I20260619 00:55:52.011467 12837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "5c86629c66604aceb8d0895b979322ce"
format_stamp: "Formatted at 2026-06-19 00:55:51 on dist-test-slave-r7bh"
server_key: "8c8bfb18527d836abd20b7100e6110c1"
server_key_iv: "83959268293fe73cd012a875cccec0f9"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:52.011883 12837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:52.050478 12837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:52.057072 12837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:52.057596 12837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:52.065580 12837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:52.074254 12837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:52.074522 12837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 00:55:52.074795 12837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:52.074967 12837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:55:52 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:52.613608 12837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36257
I20260619 00:55:52.615506 12837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:55:52.619009 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36257 every 8 connection(s)
Jun 19 00:55:52 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:52.624670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 12837
I20260619 00:55:52.624967 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:55:52.626013 388 external_mini_cluster.cc:1428] Setting key a6a1d1327857a940970a9d3a244b3aeb
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 19 00:55:52 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:52.725764 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:52.644333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60623 (local address 127.0.97.62:40225)
0619 00:55:52.684639 (+ 40306us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:52.684647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:52.684679 (+ 32us) server_negotiation.cc:407] Connection header received
0619 00:55:52.684758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:52.684763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:52.693630 (+ 8867us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:52.693768 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:52.699900 (+ 6132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:52.701274 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:52.704284 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:52.705127 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:52.705276 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:52.715074 (+ 9798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:52.715085 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:52.715090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:52.715143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:52.719120 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:52.721105 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:52.721113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:52.721117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:52.721302 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:52.721928 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:52.721934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:52.721937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:52.722353 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:52.722500 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:52.725067 (+ 2567us) server_negotiation.cc:299] Negotiation successful
0619 00:55:52.725332 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40134,"thread_start_us":39975,"threads_started":1}
I20260619 00:55:52.727205 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:52.645293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.1:60623)
0619 00:55:52.674107 (+ 28814us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:52.674185 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:52.676779 (+ 2594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:52.696148 (+ 19369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:52.696164 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:52.698350 (+ 2186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:52.699581 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:52.699621 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:52.701509 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:52.701518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:52.704006 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:52.704042 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:52.705642 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:52.705652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:52.706014 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:52.709884 (+ 3870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:52.709931 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:52.714694 (+ 4763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:52.720073 (+ 5379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:52.720084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:52.720105 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:52.720906 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:52.721452 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:52.721458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:52.721461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:52.721775 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:52.722630 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:52.722642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:52.723133 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 00:55:52.723471 (+ 338us) client_negotiation.cc:240] Negotiation successful
0619 00:55:52.724091 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28419,"thread_start_us":28134,"threads_started":1}
I20260619 00:55:52.729673 12971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:52.730046 12971 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:52.731051 12971 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:52.734360 12757 ts_manager.cc:194] Registered new tserver with Master: 5c86629c66604aceb8d0895b979322ce (127.0.97.1:36257)
I20260619 00:55:52.736650 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260619 00:55:52.740906 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.111s user 0.008s sys 0.009s
I20260619 00:55:52.741940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:55:53.180087 12978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:53.180382 12978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:53.180544 12978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:53.188038 12978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:53.188159 12978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:53.188345 12978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:55:53.198501 12978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:53.199885 12978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:53.201298 12978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:53.215379 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:53.215423 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:53.215404 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:53.216212 12978 server_base.cc:1034] running on GCE node
I20260619 00:55:53.216945 12978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:53.217957 12978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:53.219202 12978 hybrid_clock.cc:648] HybridClock initialized: now 1781830553219134 us; error 69 us; skew 500 ppm
I20260619 00:55:53.219525 12978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:53.222942 12978 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:55:53.224514 12978 webserver.cc:460] Webserver started at http://127.0.97.2:39645/ using document root <none> and password file <none>
I20260619 00:55:53.225284 12978 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:53.225394 12978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:53.225656 12978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:53.245875 12978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "54bd0cf64db14c9da93c5d87697824e9"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "8e5d118ecea27dd0cf05c2dacaabe0a6"
server_key_iv: "cc7b99f15844aed70ffbc7c4295a03c6"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.246776 12978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "54bd0cf64db14c9da93c5d87697824e9"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "8e5d118ecea27dd0cf05c2dacaabe0a6"
server_key_iv: "cc7b99f15844aed70ffbc7c4295a03c6"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.252724 12978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 00:55:53.276944 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.278934 12978 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:55:53.279143 12978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "54bd0cf64db14c9da93c5d87697824e9"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "8e5d118ecea27dd0cf05c2dacaabe0a6"
server_key_iv: "cc7b99f15844aed70ffbc7c4295a03c6"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.279412 12978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:53.304898 12978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:53.310276 12978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:53.310670 12978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:53.312003 12978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:53.313840 12978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:53.313936 12978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.314121 12978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:53.314186 12978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.358192 12978 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42421
I20260619 00:55:53.358248 13112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42421 every 8 connection(s)
I20260619 00:55:53.359520 12978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:55:53.368461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 12978
I20260619 00:55:53.368652 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:55:53.369153 388 external_mini_cluster.cc:1428] Setting key a4773ba4e48857fae52fe8f0e081ca8c
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:53.383373 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:53.362406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35533 (local address 127.0.97.62:40225)
0619 00:55:53.362921 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:53.362928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:53.364133 (+ 1205us) server_negotiation.cc:407] Connection header received
0619 00:55:53.367023 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:53.367028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:53.367105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:53.367200 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:53.369317 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.370226 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.371942 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.372410 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.372548 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:53.376000 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:53.376005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:53.376009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:53.376056 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:53.379639 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.380512 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.380524 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.380526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.380682 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.381197 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.381208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.381210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.381495 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:53.381651 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:53.382511 (+ 860us) server_negotiation.cc:299] Negotiation successful
0619 00:55:53.382800 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260619 00:55:53.384090 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:53.363006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.2:35533)
0619 00:55:53.363928 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:53.363991 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:53.366663 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:53.367521 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:53.367539 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:53.368194 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:53.369088 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:53.369116 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.370381 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.370386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:53.371792 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:53.371804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.372644 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.372651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:53.372965 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:53.373689 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:53.373712 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:53.375753 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:53.379856 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:53.379865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:53.379879 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:53.380370 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:53.380798 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:53.380804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:53.380835 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:53.381061 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:53.381622 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:53.381631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:53.382015 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 00:55:53.382321 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 00:55:53.382747 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
I20260619 00:55:53.385847 13113 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:53.386269 13113 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:53.387379 13113 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:53.389420 12757 ts_manager.cc:194] Registered new tserver with Master: 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421)
I20260619 00:55:53.390808 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35533
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:55:53.424352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:55:53.741364 12971 heartbeater.cc:503] Master 127.0.97.62:40225 was elected leader, sending a full tablet report...
W20260619 00:55:53.759172 13119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:53.759471 13119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:53.759624 13119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:53.767349 13119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:53.767477 13119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:53.767652 13119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:55:53.777807 13119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:53.779160 13119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:53.780495 13119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:53.794234 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:53.794272 13127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:53.794307 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:53.795138 13119 server_base.cc:1034] running on GCE node
I20260619 00:55:53.795881 13119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:53.796952 13119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:53.798156 13119 hybrid_clock.cc:648] HybridClock initialized: now 1781830553798114 us; error 41 us; skew 500 ppm
I20260619 00:55:53.798491 13119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:53.801853 13119 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:55:53.803308 13119 webserver.cc:460] Webserver started at http://127.0.97.3:34309/ using document root <none> and password file <none>
I20260619 00:55:53.804008 13119 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:53.804108 13119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:55:53.804371 13119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:53.821388 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "218c9e4bdec048588f397d00c5ec5e80"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "724de85f84a02836a05b418b9790bd12"
server_key_iv: "d936617d0d3cd4a417be59d7f7520cde"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.822330 13119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "218c9e4bdec048588f397d00c5ec5e80"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "724de85f84a02836a05b418b9790bd12"
server_key_iv: "d936617d0d3cd4a417be59d7f7520cde"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.828550 13119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 00:55:53.846372 13134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.848119 13119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:55:53.848328 13119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "218c9e4bdec048588f397d00c5ec5e80"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "724de85f84a02836a05b418b9790bd12"
server_key_iv: "d936617d0d3cd4a417be59d7f7520cde"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:53.848587 13119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:53.869242 13119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:53.873648 13119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:53.873960 13119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:53.874958 13119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:53.876361 13119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:53.876437 13119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.876580 13119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:53.876636 13119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:53.911839 13119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46789
I20260619 00:55:53.911861 13246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46789 every 8 connection(s)
I20260619 00:55:53.913228 13119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:55:53.918008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13119
I20260619 00:55:53.918200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:55:53.918675 388 external_mini_cluster.cc:1428] Setting key 5867c275ae8a021c8a716ba1bdba9738
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:53.936692 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:53.916072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33901 (local address 127.0.97.62:40225)
0619 00:55:53.916566 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:53.916570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:53.917678 (+ 1108us) server_negotiation.cc:407] Connection header received
0619 00:55:53.920234 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:53.920241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:53.920363 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:53.920468 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:53.922538 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.923619 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.926048 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.926515 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.926663 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:53.930226 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:53.930232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:53.930235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:53.930271 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:53.933215 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.934247 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.934252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.934254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.934393 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.934962 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.934968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.934971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.935325 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:53.935454 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:53.936194 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 00:55:53.936416 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260619 00:55:53.937738 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:53.916677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.3:33901)
0619 00:55:53.917445 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:53.917504 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:53.919870 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:53.920848 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:53.920867 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:53.921450 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:53.922342 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:53.922369 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.923848 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.923852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:53.925870 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:53.925884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.926784 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.926792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:53.927060 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:53.927783 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:53.927808 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:53.929936 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:53.933387 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:53.933397 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:53.933418 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:53.934097 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:53.934526 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:53.934530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:53.934532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:53.934807 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:53.935451 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:53.935458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:53.935814 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 00:55:53.936062 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 00:55:53.936489 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":356,"threads_started":1}
I20260619 00:55:53.939248 13247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:53.939570 13247 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:53.940322 13247 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:53.941874 12757 ts_manager.cc:194] Registered new tserver with Master: 218c9e4bdec048588f397d00c5ec5e80 (127.0.97.3:46789)
I20260619 00:55:53.942705 12757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33901
I20260619 00:55:53.947702 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:55:53.949522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 12837
Jun 19 00:55:53 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:55:53.967499 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:53.950515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.97.1:36257)
0619 00:55:53.951102 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:53.951109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:53.951136 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:55:53.951194 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:53.951201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:53.951407 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:53.951531 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:53.953248 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.954405 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.956039 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:53.956663 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:53.956762 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:53.961014 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:53.961026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:53.961040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:53.961077 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:53.964201 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.965138 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.965146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.965148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.965297 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:53.965718 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:53.965722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:53.965724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:53.966125 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:53.966385 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:53.966796 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 00:55:53.967020 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":305,"threads_started":1}
I20260619 00:55:53.968904 12965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:54.051990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 12837
I20260619 00:55:54.068300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 12978
Jun 19 00:55:54 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:55:54.090780 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:54.069343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.0.97.2:42421)
0619 00:55:54.069893 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:54.069900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:54.069925 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:55:54.069982 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:54.069988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:54.070205 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:54.070377 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:54.071820 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:54.073021 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:54.074400 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:54.074979 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:54.075135 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:54.084332 (+ 9197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:54.084349 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:54.084366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:54.084412 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:54.087776 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:54.088580 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:54.088588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:54.088590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:54.088750 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:54.089238 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:54.089242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:54.089244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:54.089616 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:54.089853 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:54.090251 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 00:55:54.090475 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260619 00:55:54.092128 13108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:54.176328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 12978
I20260619 00:55:54.192365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13119
Jun 19 00:55:54 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:55:54.209002 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:54.193457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.0.97.3:46789)
0619 00:55:54.194192 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:54.194202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:54.194236 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:55:54.194300 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:54.194306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:54.194549 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:54.194707 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:54.195946 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:54.197151 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:54.198446 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:54.198982 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:54.199074 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:54.203401 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:54.203412 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:54.203425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:54.203459 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:54.206338 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:54.207013 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:54.207020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:54.207023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:54.207165 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:54.207480 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:54.207483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:54.207486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:54.207834 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:54.208075 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:54.208322 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 00:55:54.208528 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":378,"threads_started":1}
I20260619 00:55:54.210474 13242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:54.296428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13119
I20260619 00:55:54.313005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 12725
I20260619 00:55:54.314424 12787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:54.422736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 12725
I20260619 00:55:54.444669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40225
--webserver_interface=127.0.97.62
--webserver_port=37495
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40225
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:55:54.794070 13259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:54.794397 13259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:54.794492 13259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:54.802874 13259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:55:54.803016 13259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:54.803063 13259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:54.803112 13259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:55:54.803160 13259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:55:54.814441 13259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40225
--ranger_config_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:54.815893 13259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:54.817447 13259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:54.830193 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:54.830140 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:54.830212 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:54.831337 13259 server_base.cc:1034] running on GCE node
I20260619 00:55:54.832149 13259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:54.833810 13259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:54.835062 13259 hybrid_clock.cc:648] HybridClock initialized: now 1781830554834986 us; error 60 us; skew 500 ppm
I20260619 00:55:54.835554 13259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:54 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:54.839181 13259 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:55:54.841022 13259 webserver.cc:460] Webserver started at http://127.0.97.62:37495/ using document root <none> and password file <none>
I20260619 00:55:54.841926 13259 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:54.842114 13259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 00:55:54 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:54.931497 13259 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260619 00:55:54.935067 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:54.936743 13259 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 00:55:54.937017 13259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "95e23bd94c7c4f398fe73b6f9a645c76"
format_stamp: "Formatted at 2026-06-19 00:55:50 on dist-test-slave-r7bh"
server_key: "842ff402213a2da4682329f524f3dc3e"
server_key_iv: "2d6f9ab6b56afed30947c7e6a21819f8"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:54.938145 13259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:54.953092 13259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:54.957489 13259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:54.957742 13259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:54.977388 13259 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40225
I20260619 00:55:54.977425 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40225 every 8 connection(s)
I20260619 00:55:54.979049 13259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:55:54.986794 13327 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:55:54.989274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13259
I20260619 00:55:54.990195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36257
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40225
--webserver_port=45249
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:55:54.990823 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Bootstrap starting.
I20260619 00:55:55.011236 13327 log.cc:824] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:55.021124 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:55:55.021711 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Bootstrap complete.
I20260619 00:55:55.031570 13327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:55.032536 13327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e23bd94c7c4f398fe73b6f9a645c76, State: Initialized, Role: FOLLOWER
I20260619 00:55:55.033314 13327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:55.033577 13327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:55:55.033725 13327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:55:55.033915 13327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:55:55.038156 13327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:55.038756 13327 leader_election.cc:302] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e23bd94c7c4f398fe73b6f9a645c76; no voters:
I20260619 00:55:55.039453 13327 leader_election.cc:288] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 00:55:55.039624 13331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 00:55:55.040915 13331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [term 2 LEADER]: Becoming Leader. State: Replica: 95e23bd94c7c4f398fe73b6f9a645c76, State: Running, Role: LEADER
I20260619 00:55:55.041592 13331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } }
I20260619 00:55:55.041998 13327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:55:55.044400 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } } }
I20260619 00:55:55.044855 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: This master's current role is: LEADER
I20260619 00:55:55.049623 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e23bd94c7c4f398fe73b6f9a645c76. Latest consensus state: current_term: 2 leader_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e23bd94c7c4f398fe73b6f9a645c76" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40225 } } }
I20260619 00:55:55.049964 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76 [sys.catalog]: This master's current role is: LEADER
I20260619 00:55:55.053467 13336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:55:55.060958 13336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:55:55.062844 13336 catalog_manager.cc:1259] Loaded cluster ID: 526f4c37eef14eaa9fb6927e8868d5f6
I20260619 00:55:55.062976 13336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:55:55.068491 13336 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:55:55.070699 13336 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Loaded TSK: 0
I20260619 00:55:55.072726 13336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:55:55.075896 13327 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 00:55:55.460359 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:55.460847 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:55.461045 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:55.473645 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:55.473821 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:55.474025 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:55:55.491199 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45249
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:55.493167 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:55.495050 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:55.525975 13364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:55.533227 13365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:55.533830 13329 server_base.cc:1034] running on GCE node
W20260619 00:55:55.534965 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:55.535945 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:55.559053 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:55.566717 13329 hybrid_clock.cc:648] HybridClock initialized: now 1781830555566597 us; error 120 us; skew 500 ppm
I20260619 00:55:55.567269 13329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:55 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:55.578090 13329 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:55:55.603195 13329 webserver.cc:460] Webserver started at http://127.0.97.1:45249/ using document root <none> and password file <none>
I20260619 00:55:55.604351 13329 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:55.604652 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 00:55:55 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:55.654814 13329 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.005s
I20260619 00:55:55.658958 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:55.660725 13329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 00:55:55.660990 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "5c86629c66604aceb8d0895b979322ce"
format_stamp: "Formatted at 2026-06-19 00:55:51 on dist-test-slave-r7bh"
server_key: "8c8bfb18527d836abd20b7100e6110c1"
server_key_iv: "83959268293fe73cd012a875cccec0f9"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:55.662191 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:55.695955 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:55.700645 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:55.700973 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:55.702090 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:55.704195 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:55.704324 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:55.704461 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:55.704563 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:55.783651 13329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36257
I20260619 00:55:55.783701 13488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36257 every 8 connection(s)
I20260619 00:55:55.785696 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:55:55.793464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13329
I20260619 00:55:55.794453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42421
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40225
--webserver_port=39645
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:55:55 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:55.852623 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:55.792001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.1:50355)
0619 00:55:55.815942 (+ 23941us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:55.816006 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:55.819023 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:55.829314 (+ 10291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:55.829334 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:55.830183 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:55.831266 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:55.831317 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:55.835032 (+ 3715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:55.835038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:55.837196 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:55.837216 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:55.839123 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:55.839131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:55.839407 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:55.840236 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:55.840265 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:55.843255 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:55.847662 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:55.847671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:55.847688 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:55.848240 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:55.848781 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:55.848785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:55.848788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:55.849055 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:55.850134 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:55.850141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:55.850500 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 00:55:55.850801 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 00:55:55.851349 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23543,"thread_start_us":28370,"threads_started":1}
I20260619 00:55:55.852910 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:55.793232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50355 (local address 127.0.97.62:40225)
0619 00:55:55.813410 (+ 20178us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:55.813442 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:55.819552 (+ 6110us) server_negotiation.cc:407] Connection header received
0619 00:55:55.820039 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:55.820060 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:55.820616 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:55.821067 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:55.832355 (+ 11288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:55.834738 (+ 2383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:55.837388 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:55.838463 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:55.838631 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:55.843574 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:55.843585 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:55.843606 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:55.843656 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:55.847480 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:55.848400 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:55.848411 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:55.848419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:55.848618 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:55.849185 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:55.849190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:55.849194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:55.849846 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:55.850024 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:55.851120 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0619 00:55:55.851563 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19859,"thread_start_us":19627,"threads_started":1}
I20260619 00:55:55.856353 13489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:55.856683 13489 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:55.857560 13489 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
W20260619 00:55:55.860862 13489 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40225 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 00:55:55.861099 13489 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:55.861380 13489 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:55.862772 13489 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:55.863121 13489 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
W20260619 00:55:55.864252 13489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 00:55:56.265195 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:56.265596 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:56.265753 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:56.277647 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:56.277819 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:56.278003 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:55:56.294652 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39645
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:56.296411 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:56.298183 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 19 00:55:56 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 00:55:56.331037 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:56.332062 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:56.336226 13493 server_base.cc:1034] running on GCE node
W20260619 00:55:56.336350 13512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:56.337195 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:56.338383 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:56.339622 13493 hybrid_clock.cc:648] HybridClock initialized: now 1781830556339551 us; error 68 us; skew 500 ppm
I20260619 00:55:56.340009 13493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:56 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:56.343852 13493 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:55:56.345935 13493 webserver.cc:460] Webserver started at http://127.0.97.2:39645/ using document root <none> and password file <none>
I20260619 00:55:56.346776 13493 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:56.346933 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 00:55:56 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:56.383746 13493 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.004s
I20260619 00:55:56.387835 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:56.389683 13493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 00:55:56.389886 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "54bd0cf64db14c9da93c5d87697824e9"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "8e5d118ecea27dd0cf05c2dacaabe0a6"
server_key_iv: "cc7b99f15844aed70ffbc7c4295a03c6"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:56.390970 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:56.419168 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:56.423804 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:56.424135 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:56.425376 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:56.427300 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:56.427419 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:56.427527 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:56.427603 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:56.510550 13493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42421
I20260619 00:55:56.510640 13633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42421 every 8 connection(s)
I20260619 00:55:56.512393 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:55:56.519972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13493
I20260619 00:55:56.520776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46789
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40225
--webserver_port=34309
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:55:56 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:56.549708 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:56.516924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53855 (local address 127.0.97.62:40225)
0619 00:55:56.525351 (+ 8427us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:56.525358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:56.529246 (+ 3888us) server_negotiation.cc:407] Connection header received
0619 00:55:56.529325 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:56.529331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:56.529437 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:56.529541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:56.531787 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:56.533063 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:56.535704 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:56.536340 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:56.536450 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:56.541688 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:56.541696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:56.541701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:56.541747 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:56.545478 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:56.546499 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:56.546508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:56.546515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:56.546669 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:56.547272 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:56.547280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:56.547286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:56.547692 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:56.547821 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:56.548918 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0619 00:55:56.549200 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8262,"thread_start_us":8387,"threads_started":1}
I20260619 00:55:56.552536 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:56.516193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.2:53855)
0619 00:55:56.525958 (+ 9765us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:56.526027 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:56.528635 (+ 2608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:56.530026 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:56.530039 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:56.530703 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:56.531559 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:56.531586 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:56.533311 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:56.533316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:56.535502 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:56.535518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:56.536878 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:56.536887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:56.537198 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:56.538057 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:56.538090 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:56.540958 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:56.545656 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:56.545668 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:56.545691 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:56.546358 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:56.546826 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:56.546832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:56.546839 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:56.547126 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:56.547934 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:56.547946 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:56.548445 (+ 499us) client_negotiation.cc:769] Sending connection context
0619 00:55:56.550326 (+ 1881us) client_negotiation.cc:240] Negotiation successful
0619 00:55:56.550960 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9409,"thread_start_us":13098,"threads_started":1}
I20260619 00:55:56.554850 13634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:56.555209 13634 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:56.556176 13634 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:56.558418 13292 ts_manager.cc:194] Registered new tserver with Master: 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421)
I20260619 00:55:56.560007 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53855
Jun 19 00:55:56 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:56.864405 13489 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:56.864874 13489 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:56.866248 13292 ts_manager.cc:194] Registered new tserver with Master: 5c86629c66604aceb8d0895b979322ce (127.0.97.1:36257)
I20260619 00:55:56.867393 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50355
W20260619 00:55:56.970265 13638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:55:56.970623 13638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:55:56.970799 13638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:55:56.979324 13638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:55:56.979526 13638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:55:56.979733 13638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:55:56.994583 13638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34309
--tserver_master_addrs=127.0.97.62:40225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:55:56.996557 13638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:55:56.998389 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:55:57.017794 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:57.025166 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:55:57.026010 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:55:57.026654 13638 server_base.cc:1034] running on GCE node
I20260619 00:55:57.027784 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:55:57.030086 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:55:57.031435 13638 hybrid_clock.cc:648] HybridClock initialized: now 1781830557031350 us; error 79 us; skew 500 ppm
I20260619 00:55:57.031867 13638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:55:57 dist-test-slave-r7bh krb5kdc[12281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:55:57.036353 13638 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:55:57.057590 13638 webserver.cc:460] Webserver started at http://127.0.97.3:34309/ using document root <none> and password file <none>
I20260619 00:55:57.058665 13638 fs_manager.cc:359] Metadata directory not provided
I20260619 00:55:57.058840 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 00:55:57 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:57.086280 13638 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
Jun 19 00:55:57 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 00:55:57.096935 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:57.099184 13638 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260619 00:55:57.099501 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "218c9e4bdec048588f397d00c5ec5e80"
format_stamp: "Formatted at 2026-06-19 00:55:53 on dist-test-slave-r7bh"
server_key: "724de85f84a02836a05b418b9790bd12"
server_key_iv: "d936617d0d3cd4a417be59d7f7520cde"
server_key_version: "kuduclusterkey@0"
I20260619 00:55:57.101212 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:55:57.137038 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:55:57.146624 13638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:55:57.146984 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:55:57.165343 13638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:55:57.181835 13638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:55:57.181991 13638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:57.182121 13638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:55:57.182204 13638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:55:57.315760 13638 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46789
I20260619 00:55:57.315928 13767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46789 every 8 connection(s)
I20260619 00:55:57.317744 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:55:57.320852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13638
I20260619 00:55:57.364428 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.323493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.97.62:40225)
0619 00:55:57.325624 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.325632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.340376 (+ 14744us) server_negotiation.cc:407] Connection header received
0619 00:55:57.340618 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.340624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.340733 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.340896 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:57.343637 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.344800 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.352954 (+ 8154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.353682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.353846 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.357587 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:57.357597 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:57.357605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:57.357687 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:57.361332 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.362096 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.362100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.362103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.362232 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.362658 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.362662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.362664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.363061 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:57.363333 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.363678 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.363938 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":4841,"threads_started":1}
Jun 19 00:55:57 dist-test-slave-r7bh krb5kdc[12281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:55:57.373497 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.331891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48163 (local address 127.0.97.62:40225)
0619 00:55:57.352645 (+ 20754us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.352653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.353898 (+ 1245us) server_negotiation.cc:407] Connection header received
0619 00:55:57.356770 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.356777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.356906 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.356988 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:57.359500 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.360979 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.362914 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.363649 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.363779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.367803 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:57.367808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:57.367812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:57.367846 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:57.370381 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.371130 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.371136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.371139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.371259 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.371729 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.371736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.371739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.372017 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:57.372167 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.372895 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.373112 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":20591,"thread_start_us":20780,"threads_started":1}
I20260619 00:55:57.374583 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.331214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40225 (local address 127.0.97.3:48163)
0619 00:55:57.353411 (+ 22197us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.353488 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.356281 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.357666 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.357680 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.358304 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:55:57.359214 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.359243 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.361186 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.361192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.362775 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.362788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.363904 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.363913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.364238 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.364991 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:55:57.365024 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:55:57.367552 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:55:57.370531 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:57.370540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:57.370559 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:57.370982 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:57.371381 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:55:57.371386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:55:57.371390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:55:57.371608 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:55:57.372152 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:55:57.372160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:55:57.372516 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.372790 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.373369 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21819,"thread_start_us":25077,"threads_started":1}
I20260619 00:55:57.375990 13768 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40225
I20260619 00:55:57.376317 13768 heartbeater.cc:464] Registering TS with master...
I20260619 00:55:57.377215 13768 heartbeater.cc:511] Master 127.0.97.62:40225 requested a full tablet report, sending...
I20260619 00:55:57.378854 13292 ts_manager.cc:194] Registered new tserver with Master: 218c9e4bdec048588f397d00c5ec5e80 (127.0.97.3:46789)
I20260619 00:55:57.379855 13292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48163
I20260619 00:55:57.380002 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:55:57.397573 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.383801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.97.62:40225)
0619 00:55:57.384110 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.384118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.384353 (+ 235us) server_negotiation.cc:407] Connection header received
0619 00:55:57.384583 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.384588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.384652 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.384761 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:57.386359 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.387077 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.388430 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.388893 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.389033 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.391757 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:57.391762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:57.391765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:57.391802 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:57.394637 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.395385 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.395391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.395399 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.395542 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.395997 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.396002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.396005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.396301 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:57.396582 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.396996 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.397190 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 00:55:57.403541 13292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:55:57.412020 13292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:55:57.440135 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.427542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.0.97.3:46789)
0619 00:55:57.428961 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.428974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.430242 (+ 1268us) server_negotiation.cc:407] Connection header received
0619 00:55:57.430811 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.430825 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.431162 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.431357 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.432793 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.434375 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.436903 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.438373 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.438520 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.438771 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.439629 (+ 858us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.439803 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150,"thread_start_us":1528,"threads_started":1}
I20260619 00:55:57.440245 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.427133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36412)
0619 00:55:57.430237 (+ 3104us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.430261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.430471 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.431653 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.431660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.431695 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.432466 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.432488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.434259 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.434264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.436590 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.436602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.438980 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.438986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.439258 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.439398 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.439538 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.439746 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2977,"mutex_wait_us":764,"spinlock_wait_cycles":20608,"thread_start_us":4430,"threads_started":1}
I20260619 00:55:57.440181 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.427534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.1:36257)
0619 00:55:57.429557 (+ 2023us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.429568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.430724 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 00:55:57.430835 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.430843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.431154 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.431349 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.432719 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.434103 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.436757 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.438373 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.438507 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.438774 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.439626 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.439823 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":1613,"threads_started":1}
I20260619 00:55:57.441175 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.427133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.0.1:58840)
0619 00:55:57.430017 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.430094 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.430472 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.431698 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.431702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.431722 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.432517 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.432532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.434546 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.434550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.436641 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.436658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.438989 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.438993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.439259 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.439399 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.440709 (+ 1310us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.440879 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":69,"spinlock_wait_cycles":9728,"thread_start_us":392,"threads_started":1}
I20260619 00:55:57.441442 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.428787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.0.97.2:42421)
0619 00:55:57.429744 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.429756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.432637 (+ 2881us) server_negotiation.cc:407] Connection header received
0619 00:55:57.432940 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.432950 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.433241 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.433455 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.434271 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.435796 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.438325 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.439722 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.439843 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.440034 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.440938 (+ 904us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.441098 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":612,"threads_started":1}
I20260619 00:55:57.441936 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.428069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42421 (local address 127.0.0.1:48742)
0619 00:55:57.432499 (+ 4430us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.432515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.432680 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.433450 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.433455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.433480 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.434079 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.434091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.435957 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.435963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.438064 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.438079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.440215 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.440225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.440462 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.440501 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.440595 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.440738 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4266,"mutex_wait_us":24,"thread_start_us":1888,"threads_started":1}
I20260619 00:55:57.444799 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 8d204169fc0e41319b0b3885cc1c691a (DEFAULT_TABLE table=test-table [id=890428c25c864bbe802693d8a1518df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:57.444799 13703 tablet_service.cc:1467] Processing CreateTablet for tablet 8d204169fc0e41319b0b3885cc1c691a (DEFAULT_TABLE table=test-table [id=890428c25c864bbe802693d8a1518df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:57.446306 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d204169fc0e41319b0b3885cc1c691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:57.446306 13703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d204169fc0e41319b0b3885cc1c691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:57.445968 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 8d204169fc0e41319b0b3885cc1c691a (DEFAULT_TABLE table=test-table [id=890428c25c864bbe802693d8a1518df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:55:57.447109 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d204169fc0e41319b0b3885cc1c691a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:55:57.460850 13796 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Bootstrap starting.
I20260619 00:55:57.462155 13798 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Bootstrap starting.
I20260619 00:55:57.465185 13796 tablet_bootstrap.cc:654] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:57.465670 13797 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Bootstrap starting.
I20260619 00:55:57.466313 13798 tablet_bootstrap.cc:654] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:57.466575 13796 log.cc:824] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:57.468291 13798 log.cc:824] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:57.469285 13797 tablet_bootstrap.cc:654] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Neither blocks nor log segments found. Creating new log.
I20260619 00:55:57.470293 13797 log.cc:824] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Log is configured to *not* fsync() on all Append() calls
I20260619 00:55:57.473460 13797 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: No bootstrap required, opened a new log
I20260619 00:55:57.473789 13797 ts_tablet_manager.cc:1397] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 00:55:57.474700 13798 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: No bootstrap required, opened a new log
I20260619 00:55:57.475013 13798 ts_tablet_manager.cc:1397] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260619 00:55:57.475196 13796 tablet_bootstrap.cc:492] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: No bootstrap required, opened a new log
I20260619 00:55:57.475469 13796 ts_tablet_manager.cc:1397] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260619 00:55:57.483165 13798 raft_consensus.cc:348] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.483614 13798 raft_consensus.cc:374] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:57.483709 13798 raft_consensus.cc:729] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c86629c66604aceb8d0895b979322ce, State: Initialized, Role: FOLLOWER
I20260619 00:55:57.484398 13798 consensus_queue.cc:260] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.485071 13797 raft_consensus.cc:348] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.485531 13797 raft_consensus.cc:374] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:57.485600 13489 heartbeater.cc:503] Master 127.0.97.62:40225 was elected leader, sending a full tablet report...
I20260619 00:55:57.485618 13797 raft_consensus.cc:729] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218c9e4bdec048588f397d00c5ec5e80, State: Initialized, Role: FOLLOWER
I20260619 00:55:57.486325 13798 ts_tablet_manager.cc:1428] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260619 00:55:57.486330 13797 consensus_queue.cc:260] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.487421 13768 heartbeater.cc:503] Master 127.0.97.62:40225 was elected leader, sending a full tablet report...
I20260619 00:55:57.487330 13796 raft_consensus.cc:348] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.487779 13796 raft_consensus.cc:374] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:55:57.487880 13796 raft_consensus.cc:729] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bd0cf64db14c9da93c5d87697824e9, State: Initialized, Role: FOLLOWER
I20260619 00:55:57.488147 13797 ts_tablet_manager.cc:1428] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 00:55:57.488492 13796 consensus_queue.cc:260] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.489984 13634 heartbeater.cc:503] Master 127.0.97.62:40225 was elected leader, sending a full tablet report...
I20260619 00:55:57.490777 13796 ts_tablet_manager.cc:1428] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 00:55:57.520756 13635 tablet.cc:2369] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:55:57.578698 13769 tablet.cc:2369] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:55:57.583812 13490 tablet.cc:2369] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:55:57.858860 13802 raft_consensus.cc:482] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:55:57.859246 13802 raft_consensus.cc:504] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.859649 13803 raft_consensus.cc:482] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:55:57.859933 13803 raft_consensus.cc:504] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.861713 13802 leader_election.cc:288] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218c9e4bdec048588f397d00c5ec5e80 (127.0.97.3:46789), 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421)
I20260619 00:55:57.861847 13803 leader_election.cc:288] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c86629c66604aceb8d0895b979322ce (127.0.97.1:36257), 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421)
I20260619 00:55:57.870500 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.862145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.97.3:60039)
0619 00:55:57.862401 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.862422 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.862567 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.863766 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.863774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.863802 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.864018 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.864033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.865589 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.865594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.867770 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.867785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.869691 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.869698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.869919 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.869957 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.870046 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.870206 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":6}
I20260619 00:55:57.870587 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.862484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51747 (local address 127.0.97.2:42421)
0619 00:55:57.862698 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.862706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.863195 (+ 489us) server_negotiation.cc:407] Connection header received
0619 00:55:57.863457 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.863464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.863569 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.863628 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.864496 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.865594 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.868163 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.869509 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.869602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.869691 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.870208 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.870345 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:55:57.871119 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.863059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60039 (local address 127.0.97.1:36257)
0619 00:55:57.863274 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.863280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.863301 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:55:57.863374 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.863381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.863459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.863553 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.864217 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.865411 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.868043 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.869328 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.869472 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.869583 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.870750 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.870893 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:55:57.871440 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.862404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42421 (local address 127.0.97.3:51747)
0619 00:55:57.863083 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.863101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.863295 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.863826 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.863830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.863925 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.864322 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.864333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.866003 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.866008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.867972 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.867985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.869860 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.869865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.870032 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.870063 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.870902 (+ 839us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.871038 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"spinlock_wait_cycles":24832,"thread_start_us":387,"threads_started":1}
I20260619 00:55:57.872093 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "218c9e4bdec048588f397d00c5ec5e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c86629c66604aceb8d0895b979322ce" is_pre_election: true
I20260619 00:55:57.872706 13444 raft_consensus.cc:2454] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218c9e4bdec048588f397d00c5ec5e80 in term 0.
I20260619 00:55:57.872583 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "218c9e4bdec048588f397d00c5ec5e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bd0cf64db14c9da93c5d87697824e9" is_pre_election: true
I20260619 00:55:57.873282 13589 raft_consensus.cc:2454] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218c9e4bdec048588f397d00c5ec5e80 in term 0.
I20260619 00:55:57.873595 13656 leader_election.cc:302] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218c9e4bdec048588f397d00c5ec5e80, 5c86629c66604aceb8d0895b979322ce; no voters:
I20260619 00:55:57.874337 13803 raft_consensus.cc:2781] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:55:57.874505 13803 raft_consensus.cc:482] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:55:57.874626 13803 raft_consensus.cc:3037] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:57.877954 13803 raft_consensus.cc:504] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.879294 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "218c9e4bdec048588f397d00c5ec5e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c86629c66604aceb8d0895b979322ce"
I20260619 00:55:57.879329 13589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "218c9e4bdec048588f397d00c5ec5e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bd0cf64db14c9da93c5d87697824e9"
I20260619 00:55:57.879537 13444 raft_consensus.cc:3037] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:57.879577 13589 raft_consensus.cc:3037] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:55:57.880682 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.863054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39857 (local address 127.0.97.3:46789)
0619 00:55:57.863229 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.863234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.872683 (+ 9449us) server_negotiation.cc:407] Connection header received
0619 00:55:57.873322 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.873330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.873414 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.873486 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.874284 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.875422 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.878128 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.879351 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.879472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.879583 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.880271 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.880433 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 00:55:57.880972 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.863571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49259 (local address 127.0.97.2:42421)
0619 00:55:57.870798 (+ 7227us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.870803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.873638 (+ 2835us) server_negotiation.cc:407] Connection header received
0619 00:55:57.873952 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.873960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.874036 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.874093 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:55:57.874898 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.876105 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.878287 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.879513 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.879649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.879782 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.880449 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.880603 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7127,"thread_start_us":9373,"threads_started":1}
I20260619 00:55:57.881524 13803 leader_election.cc:288] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [CANDIDATE]: Term 1 election: Requested vote from peers 5c86629c66604aceb8d0895b979322ce (127.0.97.1:36257), 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421)
I20260619 00:55:57.881635 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.863234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.97.1:39857)
0619 00:55:57.872580 (+ 9346us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.872601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.872749 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.873685 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.873688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.873756 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.874101 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.874112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.875612 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.875617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.877923 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.877945 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.879744 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.879750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.879977 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.880028 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.880909 (+ 881us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.881066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9212,"spinlock_wait_cycles":57344,"thread_start_us":10902,"threads_started":1}
I20260619 00:55:57.882354 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.863609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42421 (local address 127.0.97.1:49259)
0619 00:55:57.873517 (+ 9908us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:57.873544 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:55:57.873771 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:55:57.874279 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:55:57.874284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:55:57.874308 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:55:57.874697 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.874711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.876330 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.876335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.878111 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:55:57.878125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.880034 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.880044 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:55:57.880257 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.880292 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:55:57.881205 (+ 913us) client_negotiation.cc:240] Negotiation successful
0619 00:55:57.881364 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9771,"mutex_wait_us":5,"spinlock_wait_cycles":5760,"thread_start_us":12977,"threads_started":1}
I20260619 00:55:57.882642 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "5c86629c66604aceb8d0895b979322ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218c9e4bdec048588f397d00c5ec5e80" is_pre_election: true
I20260619 00:55:57.883134 13723 raft_consensus.cc:2379] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c86629c66604aceb8d0895b979322ce in current term 1: Already voted for candidate 218c9e4bdec048588f397d00c5ec5e80 in this term.
I20260619 00:55:57.883134 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d204169fc0e41319b0b3885cc1c691a" candidate_uuid: "5c86629c66604aceb8d0895b979322ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bd0cf64db14c9da93c5d87697824e9" is_pre_election: true
I20260619 00:55:57.883801 13444 raft_consensus.cc:2454] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218c9e4bdec048588f397d00c5ec5e80 in term 1.
I20260619 00:55:57.884212 13589 raft_consensus.cc:2454] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218c9e4bdec048588f397d00c5ec5e80 in term 1.
I20260619 00:55:57.884203 13656 leader_election.cc:302] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218c9e4bdec048588f397d00c5ec5e80, 5c86629c66604aceb8d0895b979322ce; no voters:
I20260619 00:55:57.884220 13378 leader_election.cc:302] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c86629c66604aceb8d0895b979322ce; no voters: 218c9e4bdec048588f397d00c5ec5e80, 54bd0cf64db14c9da93c5d87697824e9
I20260619 00:55:57.884552 13803 raft_consensus.cc:2781] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:55:57.884665 13802 raft_consensus.cc:2726] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 00:55:57.885073 13803 raft_consensus.cc:686] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 LEADER]: Becoming Leader. State: Replica: 218c9e4bdec048588f397d00c5ec5e80, State: Running, Role: LEADER
I20260619 00:55:57.885562 13803 consensus_queue.cc:237] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } }
I20260619 00:55:57.890343 13292 catalog_manager.cc:5495] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218c9e4bdec048588f397d00c5ec5e80 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "218c9e4bdec048588f397d00c5ec5e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218c9e4bdec048588f397d00c5ec5e80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 } health_report { overall_health: UNKNOWN } } }
I20260619 00:55:57.910097 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.904332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.0.97.3:46789)
0619 00:55:57.904507 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.904511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.904945 (+ 434us) server_negotiation.cc:407] Connection header received
0619 00:55:57.905140 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.905144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.905222 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.905349 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:55:57.905720 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.906446 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.907985 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.908417 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.908516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.909046 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:55:57.909564 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:55:57.909691 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.909796 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.909863 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:55:57.918253 13589 raft_consensus.cc:1261] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 1 FOLLOWER]: Refusing update from remote peer 218c9e4bdec048588f397d00c5ec5e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:55:57.918298 13444 raft_consensus.cc:1261] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce [term 1 FOLLOWER]: Refusing update from remote peer 218c9e4bdec048588f397d00c5ec5e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:55:57.919224 13809 consensus_queue.cc:1035] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bd0cf64db14c9da93c5d87697824e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:55:57.919574 13803 consensus_queue.cc:1035] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c86629c66604aceb8d0895b979322ce" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:55:57.932562 13815 mvcc.cc:205] Tried to move back new op lower bound from 7298377965223231488 to 7298377965102907392. Current Snapshot: MvccSnapshot[applied={T|T < 7298377965223231488}]
I20260619 00:55:57.935038 13813 mvcc.cc:205] Tried to move back new op lower bound from 7298377965223231488 to 7298377965102907392. Current Snapshot: MvccSnapshot[applied={T|T < 7298377965223231488}]
I20260619 00:55:57.959256 13292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:55:57.959586 13292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:55:57.964092 13292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95e23bd94c7c4f398fe73b6f9a645c76: Sending DeleteTablet for 3 replicas of tablet 8d204169fc0e41319b0b3885cc1c691a
I20260619 00:55:57.965543 13703 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d204169fc0e41319b0b3885cc1c691a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260619 00:55:57.966110 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d204169fc0e41319b0b3885cc1c691a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36412
I20260619 00:55:57.966351 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d204169fc0e41319b0b3885cc1c691a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48742
I20260619 00:55:57.968351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13329
I20260619 00:55:57.992244 13825 tablet_replica.cc:330] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: stopping tablet replica
I20260619 00:55:57.993007 13825 raft_consensus.cc:2229] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:55:57.993666 13825 raft_consensus.cc:2258] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:55:57.994347 13823 tablet_replica.cc:330] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: stopping tablet replica
I20260619 00:55:57.994552 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:57.969313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.0.97.1:36257)
0619 00:55:57.969680 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:57.969690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:57.971747 (+ 2057us) server_negotiation.cc:407] Connection header received
0619 00:55:57.972045 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:57.972055 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:57.972186 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:57.972282 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:57.975066 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.976597 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.979188 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:57.980013 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:57.980157 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:57.985225 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:57.985256 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:57.985283 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:57.985350 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:57.990203 (+ 4853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.991528 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.991544 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.991551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.991756 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:57.992300 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:57.992305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:57.992309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:57.992799 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:57.993159 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:57.993792 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 00:55:57.994071 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 00:55:57.996059 13484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:57.998025 13825 ts_tablet_manager.cc:1905] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:55:57.996060 13823 raft_consensus.cc:2229] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:55:57.999212 13823 raft_consensus.cc:2258] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:55:58.008435 13823 ts_tablet_manager.cc:1905] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:55:58.012009 13825 ts_tablet_manager.cc:1918] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:55:58.013887 13825 log.cc:1196] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/8d204169fc0e41319b0b3885cc1c691a
I20260619 00:55:58.015437 13825 ts_tablet_manager.cc:1939] T 8d204169fc0e41319b0b3885cc1c691a P 54bd0cf64db14c9da93c5d87697824e9: Deleting consensus metadata
I20260619 00:55:58.017436 13823 ts_tablet_manager.cc:1918] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:55:58.017652 13823 log.cc:1196] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/8d204169fc0e41319b0b3885cc1c691a
I20260619 00:55:58.017738 13277 catalog_manager.cc:4841] TS 54bd0cf64db14c9da93c5d87697824e9 (127.0.97.2:42421): tablet 8d204169fc0e41319b0b3885cc1c691a (table test-table [id=890428c25c864bbe802693d8a1518df1]) successfully deleted
I20260619 00:55:58.018208 13823 ts_tablet_manager.cc:1939] T 8d204169fc0e41319b0b3885cc1c691a P 218c9e4bdec048588f397d00c5ec5e80: Deleting consensus metadata
I20260619 00:55:58.020157 13278 catalog_manager.cc:4841] TS 218c9e4bdec048588f397d00c5ec5e80 (127.0.97.3:46789): tablet 8d204169fc0e41319b0b3885cc1c691a (table test-table [id=890428c25c864bbe802693d8a1518df1]) successfully deleted
I20260619 00:55:58.110069 13824 tablet_replica.cc:330] T 8d204169fc0e41319b0b3885cc1c691a P 5c86629c66604aceb8d0895b979322ce: stopping tablet replica
I20260619 00:55:58.110332 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13329
W20260619 00:55:58.134582 13276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36257 (error 108)
I20260619 00:55:58.135196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13493
I20260619 00:55:58.137650 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.136665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36432)
0619 00:55:58.137260 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.137402 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
W20260619 00:55:58.138113 13276 catalog_manager.cc:4584] TS 5c86629c66604aceb8d0895b979322ce (127.0.97.1:36257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d204169fc0e41319b0b3885cc1c691a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
I20260619 00:55:58.152213 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.136170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.97.2:42421)
0619 00:55:58.136397 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:58.136406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:58.136575 (+ 169us) server_negotiation.cc:407] Connection header received
0619 00:55:58.136764 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:58.136774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:58.136946 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:58.137070 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:58.138910 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:58.139631 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:58.141182 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:58.141693 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:58.141832 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:58.145679 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:58.145696 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:58.145712 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:58.145752 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:58.149169 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:58.150149 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:58.150160 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:58.150164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:58.150315 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:58.150734 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:58.150737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:58.150739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:58.151091 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:58.151345 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:58.151688 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 00:55:58.151888 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:55:58.153009 13629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:58.192204 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.191596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36434)
0619 00:55:58.191894 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.192024 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 00:55:58.193533 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.193084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36448)
0619 00:55:58.193256 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.193349 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:55:58.245678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13493
I20260619 00:55:58.254385 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.253931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36460)
0619 00:55:58.254132 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.254232 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 00:55:58.255620 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.255232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36476)
0619 00:55:58.255390 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.255471 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 00:55:58.268802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13638
I20260619 00:55:58.285614 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.269776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.0.97.3:46789)
0619 00:55:58.270055 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 00:55:58.270061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:55:58.270082 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:55:58.270273 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:55:58.270279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:55:58.270390 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:55:58.270509 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:55:58.271916 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:58.272709 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:58.274341 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:55:58.274876 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:55:58.275043 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:55:58.278311 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:55:58.278326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:55:58.278341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:55:58.278391 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:55:58.282042 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:58.282990 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:58.282998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:58.283000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:58.283169 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:55:58.283651 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:55:58.283657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:55:58.283659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:55:58.284153 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:55:58.284519 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 00:55:58.284968 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 00:55:58.285253 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260619 00:55:58.286428 13763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:58.336417 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.335790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36484)
0619 00:55:58.336073 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.336221 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 00:55:58.337839 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.337292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36500)
0619 00:55:58.337543 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.337646 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 00:55:58.412300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13638
I20260619 00:55:58.427901 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13259
I20260619 00:55:58.429327 13322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:55:58.543488 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:55:58.542962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36257 (local address 127.0.0.1:36510)
0619 00:55:58.543186 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 00:55:58.543289 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 00:55:58.543495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13259
I20260619 00:55:58.566048 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 00:55:58.902113 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 00:55:58.902377 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 00:55:59.248893 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 00:55:59.249 UTC [12311] LOG: received smart shutdown request
2026-06-19 00:55:59.252 UTC [12311] LOG: background worker "logical replication launcher" (PID 12320) exited with exit code 1
2026-06-19 00:55:59.252 UTC [12315] LOG: shutting down
2026-06-19 00:55:59.420 UTC [12311] LOG: database system is shut down
2026-06-19T00:55:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54886 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:55:59 dist-test-slave-r7bh krb5kdc[13841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:55:59 dist-test-slave-r7bh krb5kdc[13841](info): set up 2 sockets
Jun 19 00:55:59 dist-test-slave-r7bh krb5kdc[13841](info): commencing operation
krb5kdc: starting...
W20260619 00:56:00.669250 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:56:00 dist-test-slave-r7bh krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:56:00.913487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:01.266029 13854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:01.266409 13854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:01.266582 13854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:01.274083 13854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:01.274283 13854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:01.274406 13854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:01.274519 13854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:01.274600 13854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:01.285140 13854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:01.286612 13854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:01.288031 13854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:01.300478 13859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:01.300350 13860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:01.300860 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:01.301998 13854 server_base.cc:1034] running on GCE node
I20260619 00:56:01.302840 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:01.304385 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:01.305655 13854 hybrid_clock.cc:648] HybridClock initialized: now 1781830561305602 us; error 49 us; skew 500 ppm
I20260619 00:56:01.306072 13854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:01 dist-test-slave-r7bh krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:01.315127 13854 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:01.317118 13854 webserver.cc:460] Webserver started at http://127.0.97.62:36239/ using document root <none> and password file <none>
I20260619 00:56:01.317932 13854 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:01.318095 13854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:01.318429 13854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:01.321846 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "862298c431a642edaaf76447d833e513"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "850217ca2fdb692ee1e27be439e8dd55"
server_key_iv: "731ddc360f14238c17e04689b02962e4"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.322760 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "862298c431a642edaaf76447d833e513"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "850217ca2fdb692ee1e27be439e8dd55"
server_key_iv: "731ddc360f14238c17e04689b02962e4"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.328922 13854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260619 00:56:01.332908 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:01.334498 13854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:56:01.334745 13854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "862298c431a642edaaf76447d833e513"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "850217ca2fdb692ee1e27be439e8dd55"
server_key_iv: "731ddc360f14238c17e04689b02962e4"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.334993 13854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:01.368526 13854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:01.373481 13854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:01.373826 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:01.393860 13854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44293
I20260619 00:56:01.393908 13920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44293 every 8 connection(s)
I20260619 00:56:01.395375 13854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:01.398106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13854
I20260619 00:56:01.398367 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:56:01.399039 388 external_mini_cluster.cc:1428] Setting key af283de005f14304cbc851ce13c2f77f
I20260619 00:56:01.399730 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:01.407924 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 00:56:01 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:01.438198 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.412757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.0.97.62:44293)
0619 00:56:01.415070 (+ 2313us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:01.415077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:01.415121 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:56:01.418024 (+ 2903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:01.418030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:01.418328 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:01.418516 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:01.420597 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.421950 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.423507 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.424340 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.424558 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.428674 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:01.428690 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:01.428713 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:01.428770 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:01.433036 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.434186 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.434199 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.434206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.434427 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.434897 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.434905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.434912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.435734 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:01.435966 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:01.436667 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 00:56:01.437235 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":2086,"spinlock_wait_cycles":20352,"thread_start_us":1786,"threads_started":1}
I20260619 00:56:01.438377 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.412757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:50040)
0619 00:56:01.413930 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.413990 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:01.417487 (+ 3497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:01.418641 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:01.418654 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:01.419691 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:01.420324 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.420359 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.422169 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.422174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.423331 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.423344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.424694 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.424705 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.424974 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.425743 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:01.425768 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:01.428354 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:01.433238 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.433257 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.433275 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.433964 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.434567 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.434570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.434573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.434775 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.435917 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:01.435923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:01.436167 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 00:56:01.436434 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 00:56:01.436994 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":498,"threads_started":1}
I20260619 00:56:01.442134 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:56:01.443538 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.443008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37226)
0619 00:56:01.443237 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.443362 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 00:56:01.443862 13870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
I20260619 00:56:01.445636 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.445240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37240)
0619 00:56:01.445399 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.445487 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 00:56:01.445866 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37261: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 00:56:01.456543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37261
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:01.495193 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } attempt: 1
I20260619 00:56:01.496551 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.495957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37254)
0619 00:56:01.496234 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.496352 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 00:56:01.497773 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.497326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37258)
0619 00:56:01.497497 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.497601 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 00:56:01.497998 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37261: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
I20260619 00:56:01.566350 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } attempt: 2
I20260619 00:56:01.567955 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.567201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37262)
0619 00:56:01.567547 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.567686 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 00:56:01.569202 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.568723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37272)
0619 00:56:01.568927 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.569033 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 00:56:01.569407 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37261: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
I20260619 00:56:01.660720 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } attempt: 3
I20260619 00:56:01.662029 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.661500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37282)
0619 00:56:01.661744 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.661858 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 00:56:01.663194 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.662753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37284)
0619 00:56:01.662932 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.663019 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 00:56:01.663439 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37261: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
W20260619 00:56:01.799539 13928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:01.799849 13928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:01.800019 13928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 00:56:01.806756 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } attempt: 4
W20260619 00:56:01.808037 13928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:01.808151 13928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:01.808276 13928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:01.808393 13928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260619 00:56:01.808398 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.807515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37296)
0619 00:56:01.808021 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.808185 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
W20260619 00:56:01.808444 13928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:01.809617 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.809231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37302)
0619 00:56:01.809381 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.809483 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 00:56:01.809847 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37261: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37261: connect: Connection refused (error 111)
I20260619 00:56:01.819558 13928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:01.821147 13928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:01.822618 13928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:01.835639 13933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:01.835687 13934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:01.835657 13936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:01.836372 13928 server_base.cc:1034] running on GCE node
I20260619 00:56:01.837479 13928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:01.839180 13928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:01.840416 13928 hybrid_clock.cc:648] HybridClock initialized: now 1781830561840366 us; error 34 us; skew 500 ppm
I20260619 00:56:01.840883 13928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:01 dist-test-slave-r7bh krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:01.844338 13928 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 00:56:01.846302 13928 webserver.cc:460] Webserver started at http://127.0.97.61:45359/ using document root <none> and password file <none>
I20260619 00:56:01.847060 13928 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:01.847174 13928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:01.847460 13928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:01.850989 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data/instance:
uuid: "267fa4f96bd94a57937b5c5108e30b15"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "c3bb193063273ffb5dafe346e0bf10f1"
server_key_iv: "e5ce0e2f95027aff0ddb3f0518080789"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.851856 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal/instance:
uuid: "267fa4f96bd94a57937b5c5108e30b15"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "c3bb193063273ffb5dafe346e0bf10f1"
server_key_iv: "e5ce0e2f95027aff0ddb3f0518080789"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.857892 13928 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 00:56:01.862105 13943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:01.863896 13928 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:56:01.864144 13928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "267fa4f96bd94a57937b5c5108e30b15"
format_stamp: "Formatted at 2026-06-19 00:56:01 on dist-test-slave-r7bh"
server_key: "c3bb193063273ffb5dafe346e0bf10f1"
server_key_iv: "e5ce0e2f95027aff0ddb3f0518080789"
server_key_version: "encryptionkey@0"
I20260619 00:56:01.864449 13928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:01.890132 13928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:01.895725 13928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:01.896065 13928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:01.915768 13928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37261
I20260619 00:56:01.915827 13994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37261 every 8 connection(s)
I20260619 00:56:01.917434 13928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
I20260619 00:56:01.920354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 13928
I20260619 00:56:01.920579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-1/wal/instance
I20260619 00:56:01.921105 388 external_mini_cluster.cc:1428] Setting key e991331a490d15d17785c96cca953adb
I20260619 00:56:01.921667 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:01.929788 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 00:56:01 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:01.954870 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.933553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.97.62:44293)
0619 00:56:01.933808 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:01.933817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:01.935170 (+ 1353us) server_negotiation.cc:407] Connection header received
0619 00:56:01.937298 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:01.937307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:01.937413 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:01.937553 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:01.939518 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.940682 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.942561 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.943014 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.943173 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.948040 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:01.948045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:01.948049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:01.948123 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:01.950985 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.951840 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.951850 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.951852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.951976 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.952436 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.952445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.952447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.952755 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:01.952989 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:01.954028 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0619 00:56:01.954378 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:56:01.956059 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.934069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:50042)
0619 00:56:01.934964 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.935021 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:01.936967 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:01.937846 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:01.937856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:01.938518 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:01.939302 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.939334 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.940954 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.940959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.942356 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.942371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.943296 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.943302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.943576 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.944416 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:01.944469 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:01.947720 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:01.951169 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.951179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.951199 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.951694 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.952100 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.952105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.952109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.952306 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.952952 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:01.952961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:01.953450 (+ 489us) client_negotiation.cc:769] Sending connection context
0619 00:56:01.953838 (+ 388us) client_negotiation.cc:240] Negotiation successful
0619 00:56:01.954408 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":410,"threads_started":1}
I20260619 00:56:01.958014 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:56:01 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 00:56:01.977001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38539
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:01.977835 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.959216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.0.97.61:37261)
0619 00:56:01.960047 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:01.960057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:01.960093 (+ 36us) server_negotiation.cc:407] Connection header received
0619 00:56:01.960159 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:01.960170 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:01.960461 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:01.960648 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:01.961519 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.963441 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.964760 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.965873 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.966072 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.971013 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:01.971028 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:01.971044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:01.971081 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:01.974197 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.975058 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.975066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.975068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.975217 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:01.975880 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:01.975888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:01.975894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:01.976360 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:01.976551 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:01.977102 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 00:56:01.977451 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":604,"thread_start_us":449,"threads_started":1}
I20260619 00:56:01.977919 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.958893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37316)
0619 00:56:01.959122 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.959143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:01.959333 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:01.960672 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:01.960677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:01.961186 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:01.961339 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.961348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.963670 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.963674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.964580 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:01.964593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:01.966078 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:01.966084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:01.966303 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:01.967000 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:01.967019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:01.970729 (+ 3710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:01.974411 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.974416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.974419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.974878 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.975462 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:01.975468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:01.975472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:01.975715 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:01.976513 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:01.976520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:01.976718 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 00:56:01.976932 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 00:56:01.977313 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260619 00:56:01.980876 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:56:01.982384 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.981846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60486)
0619 00:56:01.982074 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.982204 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 00:56:01.982735 13947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:01.984941 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:01.984436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60488)
0619 00:56:01.984634 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:01.984732 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 00:56:01.985415 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.034854 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 1
I20260619 00:56:02.036161 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.035679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60502)
0619 00:56:02.035897 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.036005 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 00:56:02.037447 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.037032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60504)
0619 00:56:02.037241 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.037319 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 00:56:02.037667 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.106003 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 2
I20260619 00:56:02.107332 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.106821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60506)
0619 00:56:02.107069 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.107171 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 00:56:02.108461 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.108100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60514)
0619 00:56:02.108261 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.108332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 00:56:02.108661 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.109105 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } attempt: 5
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:56:02.123905 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.109752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37324)
0619 00:56:02.109953 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.109977 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.110116 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.110476 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.110481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.111009 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.111233 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.111246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.112243 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.112249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.113629 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.113645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.114360 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.114366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.114572 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.115222 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.115246 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.117788 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.121227 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.121233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.121238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.121673 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.122096 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.122101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.122105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.122264 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.122835 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.122840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.123013 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.123164 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.123516 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 00:56:02.123908 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.109868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.97.61:37261)
0619 00:56:02.110042 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.110046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.110071 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:56:02.110248 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.110251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.110311 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.110431 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.111402 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.112085 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.113773 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.114209 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.114305 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.118049 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.118058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.118063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.118115 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.121022 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.121815 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.121820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.121824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.121964 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.122380 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.122385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.122390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.122705 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.122872 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.123276 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.123516 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:56:02.125113 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:56:02.126394 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.125881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60518)
0619 00:56:02.126064 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.126169 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 00:56:02.127516 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.127155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60522)
0619 00:56:02.127315 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.127399 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:56:02.127766 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.179013 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 1
I20260619 00:56:02.180343 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.179829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60524)
0619 00:56:02.180050 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.180143 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 00:56:02.181504 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.181128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60538)
0619 00:56:02.181320 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.181394 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 00:56:02.181716 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.199894 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 3
I20260619 00:56:02.201287 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.200770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60550)
0619 00:56:02.201002 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.201096 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 00:56:02.202474 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.202021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60564)
0619 00:56:02.202267 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.202344 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260619 00:56:02.202672 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.249935 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 2
I20260619 00:56:02.251099 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.250607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60578)
0619 00:56:02.250856 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.250939 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 00:56:02.252118 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.251799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60584)
0619 00:56:02.251948 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.252011 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 00:56:02.252364 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
W20260619 00:56:02.329545 14002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:02.329929 14002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:02.330103 14002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:02.337800 14002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:02.337967 14002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:02.338084 14002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:02.338189 14002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:02.338274 14002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:02.345979 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 4
I20260619 00:56:02.347288 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.346777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60598)
0619 00:56:02.347025 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.347135 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 00:56:02.348485 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.348074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60600)
0619 00:56:02.348275 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.348359 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 00:56:02.348842 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
I20260619 00:56:02.350137 14002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:37261,127.0.97.60:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:02.351481 14002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:02.352936 14002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 00:56:02.358587 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 3
I20260619 00:56:02.360186 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.359328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60616)
0619 00:56:02.359725 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.359921 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260619 00:56:02.361955 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.361243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60620)
0619 00:56:02.361633 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.361756 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
W20260619 00:56:02.362481 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38539: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38539: connect: Connection refused (error 111)
W20260619 00:56:02.365869 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:02.366318 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:02.367290 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:02.367890 14002 server_base.cc:1034] running on GCE node
I20260619 00:56:02.368785 14002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:02.370538 14002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:02.371765 14002 hybrid_clock.cc:648] HybridClock initialized: now 1781830562371712 us; error 43 us; skew 500 ppm
I20260619 00:56:02.372182 14002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:02.375663 14002 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 00:56:02.377410 14002 webserver.cc:460] Webserver started at http://127.0.97.60:41813/ using document root <none> and password file <none>
I20260619 00:56:02.378171 14002 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:02.378304 14002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:02.378648 14002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:02.381942 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data/instance:
uuid: "c541dde9b68046deaf1e15de17510f4e"
format_stamp: "Formatted at 2026-06-19 00:56:02 on dist-test-slave-r7bh"
server_key: "392d4ee2248fb8edf53dc864ebbbed62"
server_key_iv: "1e7f84ba0d49546343d33525ebb2304b"
server_key_version: "encryptionkey@0"
I20260619 00:56:02.382812 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal/instance:
uuid: "c541dde9b68046deaf1e15de17510f4e"
format_stamp: "Formatted at 2026-06-19 00:56:02 on dist-test-slave-r7bh"
server_key: "392d4ee2248fb8edf53dc864ebbbed62"
server_key_iv: "1e7f84ba0d49546343d33525ebb2304b"
server_key_version: "encryptionkey@0"
I20260619 00:56:02.388459 14002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:56:02.392616 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:02.394613 14002 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 00:56:02.394819 14002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "c541dde9b68046deaf1e15de17510f4e"
format_stamp: "Formatted at 2026-06-19 00:56:02 on dist-test-slave-r7bh"
server_key: "392d4ee2248fb8edf53dc864ebbbed62"
server_key_iv: "1e7f84ba0d49546343d33525ebb2304b"
server_key_version: "encryptionkey@0"
I20260619 00:56:02.395054 14002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:02.419768 14002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:02.424685 14002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:02.424988 14002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:02.444968 14002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38539
I20260619 00:56:02.445020 14068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38539 every 8 connection(s)
I20260619 00:56:02.446517 14002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
I20260619 00:56:02.450603 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:02.453275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14002
I20260619 00:56:02.453485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830351410501-388-0/minicluster-data/master-2/wal/instance
I20260619 00:56:02.454106 388 external_mini_cluster.cc:1428] Setting key 130764c80ea592c7df17e24ec191c748
I20260619 00:56:02.459396 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:02.480862 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.462499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.0.97.62:44293)
0619 00:56:02.463162 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.463168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.463601 (+ 433us) server_negotiation.cc:407] Connection header received
0619 00:56:02.463866 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.463876 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.463941 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.464060 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.465692 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.466714 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.468562 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.469161 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.469337 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.473456 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.473465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.473472 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.473521 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.476962 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.477986 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.477992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.477995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.478221 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.478772 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.478778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.478781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.479147 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.479307 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.480158 (+ 851us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.480429 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":323,"threads_started":1}
I20260619 00:56:02.482259 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.462434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:50046)
0619 00:56:02.463449 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.463496 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.463716 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.464134 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.464142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.465119 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.465478 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.465491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.466898 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.466902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.468380 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.468397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.469502 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.469512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.469748 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.470499 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.470534 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.473211 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.477135 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.477172 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.477190 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.477799 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.478378 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.478383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.478391 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.478643 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.479308 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.479315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.479632 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.480017 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.480635 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":43,"thread_start_us":597,"threads_started":1}
I20260619 00:56:02.482300 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.456403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.0.97.60:38539)
0619 00:56:02.457363 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.457376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.457422 (+ 46us) server_negotiation.cc:407] Connection header received
0619 00:56:02.459301 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.459327 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.460023 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.460607 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.462327 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.464325 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.466326 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.467391 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.467558 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.472931 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.472942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.472963 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.473004 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.476880 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.478007 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.478017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.478020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.478261 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.478725 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.478733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.478736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.479603 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.479854 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.481031 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.481306 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":727,"thread_start_us":503,"threads_started":1}
I20260619 00:56:02.484283 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:02.501565 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.485393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.0.97.61:37261)
0619 00:56:02.485601 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.485609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.485645 (+ 36us) server_negotiation.cc:407] Connection header received
0619 00:56:02.485837 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.485843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.485927 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.486032 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.487127 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.487895 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.489459 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.489917 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.490026 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.494943 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.494951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.494957 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.495013 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.498406 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.499235 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.499243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.499247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.499432 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.499903 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.499911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.499915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.500296 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.500445 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.500954 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.501137 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 00:56:02.501613 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.485259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37261 (local address 127.0.0.1:37326)
0619 00:56:02.485557 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.485577 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.485694 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.486089 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.486097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.486673 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.486956 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.486973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.488118 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.488122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.489291 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.489306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.490102 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.490110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.490348 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.491137 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.491157 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.494610 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.498573 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.498582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.498587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.499074 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.499581 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.499591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.499593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.499770 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.500411 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.500415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.500642 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.500839 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.501276 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 00:56:02.502765 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:56:02.503497 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.486002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.0.97.62:44293)
0619 00:56:02.486209 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.486215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.486238 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:56:02.486284 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.486288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.486354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.486430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.487875 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.488637 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.490524 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.491163 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.491269 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.496927 (+ 5658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.496933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.496937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.497013 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.500217 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.501231 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.501237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.501241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.501395 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.501920 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.501924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.501926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.502225 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.502525 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.502882 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.503145 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:56:02.518870 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.503690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60638)
0619 00:56:02.503888 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.503912 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.504026 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.504427 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.504431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.504988 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.505150 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.505168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.506470 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.506474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.507543 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.507557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.508328 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.508333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.508525 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.509215 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.509239 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.511947 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.515993 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.515998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.516002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.516577 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.517077 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.517084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.517089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.517233 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.517779 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.517784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.517928 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.518089 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.518426 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 00:56:02.518903 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.503885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.97.60:38539)
0619 00:56:02.504059 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.504067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.504108 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:56:02.504168 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.504174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.504255 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.504361 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.505303 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.506329 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.507721 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.508137 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.508268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.512187 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.512196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.512205 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.512250 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.515826 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.516730 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.516734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.516742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.516955 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.517349 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.517354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.517357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.517664 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.517821 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.518206 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.518584 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":55}
I20260619 00:56:02.522571 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.505603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.0.97.61:37261)
0619 00:56:02.505774 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.505778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.505792 (+ 14us) server_negotiation.cc:407] Connection header received
0619 00:56:02.505837 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.505840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.505896 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.505972 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.507396 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.508204 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.509616 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.510263 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.510414 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.516060 (+ 5646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.516068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.516073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.516125 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.519481 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.520334 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.520340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.520343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.520512 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.521104 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.521109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.521111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.521416 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.521663 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.522060 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.522246 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:56:02.525815 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: Bootstrap starting.
I20260619 00:56:02.528643 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:56:02.529743 14069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:02.530864 14069 log.cc:824] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:02.534559 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: No bootstrap required, opened a new log
I20260619 00:56:02.541060 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 4
I20260619 00:56:02.549067 14069 raft_consensus.cc:348] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.549549 14069 raft_consensus.cc:374] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:02.549697 14069 raft_consensus.cc:729] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c541dde9b68046deaf1e15de17510f4e, State: Initialized, Role: FOLLOWER
I20260619 00:56:02.550518 14069 consensus_queue.cc:260] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:56:02.562842 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.534502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.0.97.62:44293)
0619 00:56:02.534849 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.534857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.536312 (+ 1455us) server_negotiation.cc:407] Connection header received
0619 00:56:02.536540 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.536547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.536635 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.536758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.539689 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.540891 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.546745 (+ 5854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.547475 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.547611 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.554452 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.554460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.554465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.554516 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.558777 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.560021 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.560029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.560034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.560258 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.560741 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.560746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.560750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.561175 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.561444 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.562074 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.562385 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260619 00:56:02.569929 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.534312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.97.60:38539)
0619 00:56:02.534692 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.534698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.539055 (+ 4357us) server_negotiation.cc:407] Connection header received
0619 00:56:02.539725 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.539732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.539850 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.539946 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.542655 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.544036 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.546729 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.547542 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.547633 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.551786 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.551795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.551800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.551849 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.556322 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.557355 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.557361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.557364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.557562 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.558055 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.558060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.558063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.566684 (+ 8621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.569011 (+ 2327us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.569174 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.569476 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6771,"server-negotiator.queue_time_us":263}
I20260619 00:56:02.572760 14069 sys_catalog.cc:564] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:02.570694 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:02.573307 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:02.575909 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.541714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60662)
0619 00:56:02.542076 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.542098 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.542205 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.549963 (+ 7758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.549969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.550512 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.550751 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.550765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.553407 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.553416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.555663 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.555690 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.557473 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.557483 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.557855 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.558595 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.558645 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.563428 (+ 4783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.572415 (+ 8987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.572424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.572431 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.573160 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.573687 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.573692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.573696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.573868 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.574537 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.574546 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.574802 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.574981 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.575387 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260619 00:56:02.576910 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.542425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.0.97.60:38539)
0619 00:56:02.549495 (+ 7070us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.549502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.549531 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:56:02.549603 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.549609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.549694 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.549827 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.551330 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.553163 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.555968 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.557183 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.558423 (+ 1240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.567921 (+ 9498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.567931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.567935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.568003 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.572193 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.573357 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.573363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.573366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.573542 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.573998 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.574004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.574007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.574332 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.576189 (+ 1857us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.576332 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.576520 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":6900,"thread_start_us":6565,"threads_started":1}
I20260619 00:56:02.582943 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.533839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.0.97.61:37261)
0619 00:56:02.534031 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.534039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.541864 (+ 7825us) server_negotiation.cc:407] Connection header received
0619 00:56:02.542158 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.542166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.542258 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.542340 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.550236 (+ 7896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.551499 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.560516 (+ 9017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.561242 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.561351 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.564593 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.564599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.564602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.564641 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.568980 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.580603 (+ 11623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.580611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.580617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.580879 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.581322 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.581327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.581331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.581655 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.581948 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.582286 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.582548 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:56:02.589255 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: Bootstrap starting.
I20260619 00:56:02.596558 13921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:02.598532 13921 log.cc:824] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:02.610420 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: No bootstrap required, opened a new log
I20260619 00:56:02.600455 14002 master_runner.cc:186] Error getting master registration for 127.0.97.62:44293: OK, instance_id { permanent_uuid: "862298c431a642edaaf76447d833e513" instance_seqno: 1781830561388255 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:56:02.617605 14002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 00:56:02.618624 14095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:56:02.618789 14095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:56:02.623060 13921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.623466 13921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:02.623600 13921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862298c431a642edaaf76447d833e513, State: Initialized, Role: FOLLOWER
I20260619 00:56:02.625813 13921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.627475 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:02.627998 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:02.628901 13921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:02.648186 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } attempt: 5
Jun 19 00:56:02 dist-test-slave-r7bh krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 00:56:02.677232 14023 tablet.cc:2369] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:02.644498 13854 master_runner.cc:186] Error getting master registration for 127.0.97.61:37261: OK, instance_id { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" instance_seqno: 1781830561910012 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:56:02.688072 13854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 00:56:02.688356 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.649199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38539 (local address 127.0.0.1:60670)
0619 00:56:02.649762 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:02.649792 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:02.649969 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:02.651332 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:02.651341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:02.652120 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:02.652451 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.652471 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.661294 (+ 8823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.661306 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.663595 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:02.663629 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.665693 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.665706 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:02.666075 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.667019 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:02.667053 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:02.670693 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:02.675794 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.675804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.675811 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.684968 (+ 9157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.685722 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:02.685731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:02.685737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:02.686056 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:02.686943 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:02.686951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:02.687180 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 00:56:02.687381 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 00:56:02.687822 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368}
I20260619 00:56:02.689601 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:02.650321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.0.97.60:38539)
0619 00:56:02.650626 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:02.650636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:02.650666 (+ 30us) server_negotiation.cc:407] Connection header received
0619 00:56:02.650759 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:02.650768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:02.650886 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:02.651049 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:02.652749 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.654451 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.663989 (+ 9538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:02.665147 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:02.665355 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:02.671045 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:02.671057 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:02.671064 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:02.671123 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:02.675513 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.685250 (+ 9737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.685260 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.685266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.685524 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:02.686213 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:02.686221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:02.686227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:02.686689 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:02.686857 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:02.688797 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0619 00:56:02.689149 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 00:56:02.691949 14109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:56:02.692178 14109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:56:02.696741 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: Bootstrap starting.
I20260619 00:56:02.706180 13995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:02.707762 13995 log.cc:824] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:02.723136 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: No bootstrap required, opened a new log
I20260619 00:56:02.742327 13995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.742883 13995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:02.743036 13995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267fa4f96bd94a57937b5c5108e30b15, State: Initialized, Role: FOLLOWER
I20260619 00:56:02.743954 13995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.757768 13995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:02.759521 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:02.759964 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:02.850577 14123 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 00:56:02.868943 13928 master_runner.cc:263] No leader master found from master 267fa4f96bd94a57937b5c5108e30b15
I20260619 00:56:02.869161 13928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 00:56:02.877238 13875 tablet.cc:2369] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:02.899030 13949 tablet.cc:2369] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:02.907523 14112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:56:02.907893 14112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.909693 14112 leader_election.cc:288] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862298c431a642edaaf76447d833e513 (127.0.97.62:44293), c541dde9b68046deaf1e15de17510f4e (127.0.97.60:38539)
I20260619 00:56:02.910919 14044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "267fa4f96bd94a57937b5c5108e30b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c541dde9b68046deaf1e15de17510f4e" is_pre_election: true
I20260619 00:56:02.911597 14044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 267fa4f96bd94a57937b5c5108e30b15 in term 0.
I20260619 00:56:02.913089 13947 leader_election.cc:302] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267fa4f96bd94a57937b5c5108e30b15, c541dde9b68046deaf1e15de17510f4e; no voters:
I20260619 00:56:02.915419 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "267fa4f96bd94a57937b5c5108e30b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862298c431a642edaaf76447d833e513" is_pre_election: true
I20260619 00:56:02.916062 13896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 267fa4f96bd94a57937b5c5108e30b15 in term 0.
I20260619 00:56:02.919899 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:56:02.920078 14112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:56:02.920189 14112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:02.923607 14112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.925033 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "267fa4f96bd94a57937b5c5108e30b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862298c431a642edaaf76447d833e513"
I20260619 00:56:02.925328 13896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:02.929358 13896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 267fa4f96bd94a57937b5c5108e30b15 in term 1.
I20260619 00:56:02.930030 13946 leader_election.cc:302] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267fa4f96bd94a57937b5c5108e30b15, 862298c431a642edaaf76447d833e513; no voters:
I20260619 00:56:02.932677 14044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "267fa4f96bd94a57937b5c5108e30b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c541dde9b68046deaf1e15de17510f4e"
I20260619 00:56:02.932986 14044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:02.936685 14044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 267fa4f96bd94a57937b5c5108e30b15 in term 1.
I20260619 00:56:02.937768 14112 leader_election.cc:288] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [CANDIDATE]: Term 1 election: Requested vote from peers 862298c431a642edaaf76447d833e513 (127.0.97.62:44293), c541dde9b68046deaf1e15de17510f4e (127.0.97.60:38539)
I20260619 00:56:02.938038 14112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:02.939055 14112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [term 1 LEADER]: Becoming Leader. State: Replica: 267fa4f96bd94a57937b5c5108e30b15, State: Running, Role: LEADER
I20260619 00:56:02.939877 14112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } }
I20260619 00:56:02.947788 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267fa4f96bd94a57937b5c5108e30b15. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:02.948249 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:02.949210 14126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:02.955029 14126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:02.983327 14044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [term 1 FOLLOWER]: Refusing update from remote peer 267fa4f96bd94a57937b5c5108e30b15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:02.984635 14112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:02.993028 13896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [term 1 FOLLOWER]: Refusing update from remote peer 267fa4f96bd94a57937b5c5108e30b15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:02.994242 14112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:03.033972 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267fa4f96bd94a57937b5c5108e30b15. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.034332 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:03.036583 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267fa4f96bd94a57937b5c5108e30b15. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.037019 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:03.044448 14128 mvcc.cc:205] Tried to move back new op lower bound from 7298377985964908544 to 7298377985818005504. Current Snapshot: MvccSnapshot[applied={T|T < 7298377985964908544}]
I20260619 00:56:03.051079 14130 mvcc.cc:205] Tried to move back new op lower bound from 7298377985964908544 to 7298377985818005504. Current Snapshot: MvccSnapshot[applied={T|T < 7298377985964908544}]
I20260619 00:56:03.054075 14129 mvcc.cc:205] Tried to move back new op lower bound from 7298377985964908544 to 7298377985818005504. Current Snapshot: MvccSnapshot[applied={T|T < 7298377985964908544}]
I20260619 00:56:03.055712 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.056027 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:03.061434 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.061764 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:56:03.069344 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.069739 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:03.070149 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "267fa4f96bd94a57937b5c5108e30b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862298c431a642edaaf76447d833e513" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "267fa4f96bd94a57937b5c5108e30b15" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37261 } } peers { permanent_uuid: "c541dde9b68046deaf1e15de17510f4e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38539 } } }
I20260619 00:56:03.070436 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15 [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:03.071640 14126 catalog_manager.cc:1347] Generated new cluster ID: 62a5f21e1cc24b6694473939a9e077b6
I20260619 00:56:03.071743 14126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:03.105762 14126 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:56:03.107865 14126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:03.135318 14126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 267fa4f96bd94a57937b5c5108e30b15: Generated new TSK 0
I20260619 00:56:03.136531 14126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:56:03.621117 14095 catalog_manager.cc:1259] Loaded cluster ID: 62a5f21e1cc24b6694473939a9e077b6
I20260619 00:56:03.621276 14095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: loading cluster ID for follower catalog manager: success
I20260619 00:56:03.624027 14095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: acquiring CA information for follower catalog manager: success
I20260619 00:56:03.626060 14095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c541dde9b68046deaf1e15de17510f4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:56:03.694407 14109 catalog_manager.cc:1259] Loaded cluster ID: 62a5f21e1cc24b6694473939a9e077b6
I20260619 00:56:03.694557 14109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: loading cluster ID for follower catalog manager: success
I20260619 00:56:03.697425 14109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: acquiring CA information for follower catalog manager: success
I20260619 00:56:03.699429 14109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 862298c431a642edaaf76447d833e513: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:56:03.754002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13854
I20260619 00:56:03.755421 13916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:03.867201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13854
I20260619 00:56:03.889235 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 13928
I20260619 00:56:03.890291 13990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:03.999523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 13928
I20260619 00:56:04.020503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14002
I20260619 00:56:04.021812 14064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:04.129956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14002
2026-06-19T00:56:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4586 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:56:04 dist-test-slave-r7bh krb5kdc[14144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:56:04 dist-test-slave-r7bh krb5kdc[14144](info): set up 2 sockets
Jun 19 00:56:04 dist-test-slave-r7bh krb5kdc[14144](info): commencing operation
krb5kdc: starting...
W20260619 00:56:05.245280 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:56:05 dist-test-slave-r7bh krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:56:05.486720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39977
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:05.831277 14157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:05.831579 14157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:05.831734 14157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:05.839212 14157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:05.839331 14157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:05.839439 14157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:05.839537 14157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:05.839622 14157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:05.850191 14157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:05.851658 14157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:05.853333 14157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:05.866285 14162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:05.866355 14163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:05.866328 14165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:05.867218 14157 server_base.cc:1034] running on GCE node
I20260619 00:56:05.868086 14157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:05.869746 14157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:05.870950 14157 hybrid_clock.cc:648] HybridClock initialized: now 1781830565870884 us; error 55 us; skew 500 ppm
I20260619 00:56:05.871313 14157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:05 dist-test-slave-r7bh krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:05.874727 14157 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:05.876338 14157 webserver.cc:460] Webserver started at http://127.0.97.62:42575/ using document root <none> and password file <none>
I20260619 00:56:05.877172 14157 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:05.877315 14157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:05.877626 14157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:05.880632 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "70d4be80e759464f850cb7868c5c2835"
format_stamp: "Formatted at 2026-06-19 00:56:05 on dist-test-slave-r7bh"
server_key: "10752a5c53ab2097ea1a0d5ebf0f64b3"
server_key_iv: "85f48bdbde20a5025d38ce5005508de5"
server_key_version: "encryptionkey@0"
I20260619 00:56:05.881497 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "70d4be80e759464f850cb7868c5c2835"
format_stamp: "Formatted at 2026-06-19 00:56:05 on dist-test-slave-r7bh"
server_key: "10752a5c53ab2097ea1a0d5ebf0f64b3"
server_key_iv: "85f48bdbde20a5025d38ce5005508de5"
server_key_version: "encryptionkey@0"
I20260619 00:56:05.886871 14157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:56:05.890529 14172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:05.891992 14157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:56:05.892215 14157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "70d4be80e759464f850cb7868c5c2835"
format_stamp: "Formatted at 2026-06-19 00:56:05 on dist-test-slave-r7bh"
server_key: "10752a5c53ab2097ea1a0d5ebf0f64b3"
server_key_iv: "85f48bdbde20a5025d38ce5005508de5"
server_key_version: "encryptionkey@0"
I20260619 00:56:05.892441 14157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:05.906241 14157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:05.911010 14157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:05.911329 14157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:05.931449 14157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39977
I20260619 00:56:05.931475 14223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39977 every 8 connection(s)
I20260619 00:56:05.933089 14157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:05.937295 14224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:05.941325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14157
I20260619 00:56:05.941586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:56:05.942394 388 external_mini_cluster.cc:1428] Setting key 3a5f007679810abdc030277495254e99
I20260619 00:56:05.950820 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835: Bootstrap starting.
I20260619 00:56:05.954279 14224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:05.955374 14224 log.cc:824] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:56:05 dist-test-slave-r7bh krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:05.958488 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835: No bootstrap required, opened a new log
I20260619 00:56:05.966403 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:05.944543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.0.97.62:39977)
0619 00:56:05.945417 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:05.945431 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:05.945475 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:56:05.947186 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:05.947211 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:05.947685 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:05.948149 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:05.949800 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:05.951311 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:05.952773 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:05.953538 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:05.953745 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:05.958259 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:05.958267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:05.958285 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:05.958329 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:05.961737 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:05.962630 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:05.962641 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:05.962648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:05.962847 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:05.963269 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:05.963275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:05.963280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:05.963931 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:05.964276 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:05.964743 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 00:56:05.965273 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260619 00:56:05.967051 14224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } }
I20260619 00:56:05.967538 14224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:05.967653 14224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d4be80e759464f850cb7868c5c2835, State: Initialized, Role: FOLLOWER
I20260619 00:56:05.968317 14224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } }
I20260619 00:56:05.968638 14224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:05.968875 14224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:05.969063 14224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:05.972671 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:56:05.972635 14224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } }
I20260619 00:56:05.973341 14224 leader_election.cc:302] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d4be80e759464f850cb7868c5c2835; no voters:
I20260619 00:56:05.974054 14224 leader_election.cc:288] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:56:05.974200 14229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:05.974889 14229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [term 1 LEADER]: Becoming Leader. State: Replica: 70d4be80e759464f850cb7868c5c2835, State: Running, Role: LEADER
I20260619 00:56:05.975625 14229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } }
I20260619 00:56:05.978080 14224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:05.979061 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d4be80e759464f850cb7868c5c2835. Latest consensus state: current_term: 1 leader_uuid: "70d4be80e759464f850cb7868c5c2835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } } }
I20260619 00:56:05.979036 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d4be80e759464f850cb7868c5c2835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d4be80e759464f850cb7868c5c2835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39977 } } }
I20260619 00:56:05.979527 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:05.979666 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835 [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:05.981817 14234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:05.986996 14234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:05.996961 14234 catalog_manager.cc:1347] Generated new cluster ID: 29f0f1f078ef4ea3bb3ce634de1cf12a
I20260619 00:56:05.997107 14234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:06.015159 14234 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:56:06.017282 14234 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:06.031708 14234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d4be80e759464f850cb7868c5c2835: Generated new TSK 0
I20260619 00:56:06.032891 14234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:56:06.086637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14157
I20260619 00:56:06.087872 14219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:06.187278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14157
2026-06-19T00:56:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2051 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:56:06 dist-test-slave-r7bh krb5kdc[14252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:56:06 dist-test-slave-r7bh krb5kdc[14252](info): set up 2 sockets
Jun 19 00:56:06 dist-test-slave-r7bh krb5kdc[14252](info): commencing operation
krb5kdc: starting...
W20260619 00:56:07.289389 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.014s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:56:07 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:56:07.534593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:07.876075 14265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:07.876392 14265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:07.876566 14265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:07.884107 14265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:07.884229 14265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:07.884343 14265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:07.884460 14265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:07.884545 14265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:07.894799 14265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:07.896267 14265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:07.897701 14265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:07.910048 14270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:07.910075 14271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:07.910231 14273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:07.911020 14265 server_base.cc:1034] running on GCE node
I20260619 00:56:07.911824 14265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:07.913342 14265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:07.914582 14265 hybrid_clock.cc:648] HybridClock initialized: now 1781830567914503 us; error 38 us; skew 500 ppm
I20260619 00:56:07.914953 14265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:07 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:07.922209 14265 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:07.923882 14265 webserver.cc:460] Webserver started at http://127.0.97.62:43571/ using document root <none> and password file <none>
I20260619 00:56:07.924734 14265 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:07.924906 14265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:07.925227 14265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:07.928228 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "ab53d0096d0c408797a3fb950e03e5fb"
format_stamp: "Formatted at 2026-06-19 00:56:07 on dist-test-slave-r7bh"
server_key: "49f5923af0afb07f928e0a812199e4b8"
server_key_iv: "1f798e46a33143907c4301cecd8b0254"
server_key_version: "encryptionkey@0"
I20260619 00:56:07.929073 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "ab53d0096d0c408797a3fb950e03e5fb"
format_stamp: "Formatted at 2026-06-19 00:56:07 on dist-test-slave-r7bh"
server_key: "49f5923af0afb07f928e0a812199e4b8"
server_key_iv: "1f798e46a33143907c4301cecd8b0254"
server_key_version: "encryptionkey@0"
I20260619 00:56:07.934249 14265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 00:56:07.937906 14280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:07.939414 14265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:56:07.939633 14265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "ab53d0096d0c408797a3fb950e03e5fb"
format_stamp: "Formatted at 2026-06-19 00:56:07 on dist-test-slave-r7bh"
server_key: "49f5923af0afb07f928e0a812199e4b8"
server_key_iv: "1f798e46a33143907c4301cecd8b0254"
server_key_version: "encryptionkey@0"
I20260619 00:56:07.939867 14265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:07.969985 14265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:07.974521 14265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:07.974846 14265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:07.997380 14265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37873
I20260619 00:56:07.997390 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37873 every 8 connection(s)
I20260619 00:56:07.998886 14265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:08.003098 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:08.008652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14265
I20260619 00:56:08.008929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:56:08.009821 388 external_mini_cluster.cc:1428] Setting key 63dfb810da859a55b8a420ab0bb3ce92
I20260619 00:56:08.016196 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap starting.
I20260619 00:56:08.019919 14332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:08.021492 14332 log.cc:824] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:08.025290 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: No bootstrap required, opened a new log
Jun 19 00:56:08 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:08.034723 14332 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:08.035164 14332 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:08.035285 14332 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Initialized, Role: FOLLOWER
I20260619 00:56:08.035924 14332 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:08.036190 14332 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:08.036355 14332 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:08.036558 14332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:08.039621 14332 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:08.040169 14332 leader_election.cc:302] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab53d0096d0c408797a3fb950e03e5fb; no voters:
I20260619 00:56:08.040928 14332 leader_election.cc:288] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:56:08.041066 14337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:08.041453 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:08.012454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.62:37873)
0619 00:56:08.016994 (+ 4540us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:08.017008 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:08.017058 (+ 50us) server_negotiation.cc:407] Connection header received
0619 00:56:08.018936 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:08.018969 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:08.019436 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:08.019854 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:08.021850 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.023872 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.025871 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.026604 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.026755 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:08.031958 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:08.031972 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:08.031989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:08.032043 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:08.036079 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:08.037140 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:08.037147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:08.037155 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:08.037370 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:08.037856 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:08.037860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:08.037862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:08.038532 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:08.038783 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:08.039264 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 00:56:08.039743 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4259,"thread_start_us":3946,"threads_started":1}
I20260619 00:56:08.043284 14332 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:08.048054 14337 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 LEADER]: Becoming Leader. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Running, Role: LEADER
I20260619 00:56:08.048864 14337 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:08.052335 14339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab53d0096d0c408797a3fb950e03e5fb. Latest consensus state: current_term: 1 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:08.052753 14339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:08.052163 14338 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:08.054061 14338 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
W20260619 00:56:08.065754 14351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:56:08.065886 14351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:56:08.066027 14352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:08.068190 14352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:08.077533 14352 catalog_manager.cc:1347] Generated new cluster ID: 592d6140578a4be9b73a659b607c7ee5
I20260619 00:56:08.077677 14352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:08.091434 14352 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:56:08.093397 14352 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:08.108489 14352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Generated new TSK 0
I20260619 00:56:08.109783 14352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:56:08.164197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:08.512318 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:08.512620 14358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:08.512714 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:08.520282 14358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:08.520414 14358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:08.520531 14358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:56:08.530999 14358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:08.532368 14358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:08.533762 14358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:08.548945 14363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:08.548971 14364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:08.549067 14366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:08.549729 14358 server_base.cc:1034] running on GCE node
I20260619 00:56:08.550462 14358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:08.551628 14358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:08.552929 14358 hybrid_clock.cc:648] HybridClock initialized: now 1781830568552875 us; error 45 us; skew 500 ppm
I20260619 00:56:08.553293 14358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:08 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:08.556768 14358 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:56:08.558460 14358 webserver.cc:460] Webserver started at http://127.0.97.1:42005/ using document root <none> and password file <none>
I20260619 00:56:08.559165 14358 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:08.559271 14358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:08.559530 14358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:08.562878 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "1cc5587acbd748d7ade71413bb09a48b"
format_stamp: "Formatted at 2026-06-19 00:56:08 on dist-test-slave-r7bh"
server_key: "25513c71ba9d259fedbe8c9b454b7cea"
server_key_iv: "25b5235e0db9b6db754ceb7b2d02be4c"
server_key_version: "encryptionkey@0"
I20260619 00:56:08.563673 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1cc5587acbd748d7ade71413bb09a48b"
format_stamp: "Formatted at 2026-06-19 00:56:08 on dist-test-slave-r7bh"
server_key: "25513c71ba9d259fedbe8c9b454b7cea"
server_key_iv: "25b5235e0db9b6db754ceb7b2d02be4c"
server_key_version: "encryptionkey@0"
I20260619 00:56:08.569291 14358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:56:08.572916 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:08.574493 14358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:56:08.574689 14358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "1cc5587acbd748d7ade71413bb09a48b"
format_stamp: "Formatted at 2026-06-19 00:56:08 on dist-test-slave-r7bh"
server_key: "25513c71ba9d259fedbe8c9b454b7cea"
server_key_iv: "25b5235e0db9b6db754ceb7b2d02be4c"
server_key_version: "encryptionkey@0"
I20260619 00:56:08.574921 14358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:08.598510 14358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:08.603392 14358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:08.603858 14358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:08.605167 14358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:08.606719 14358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:08.606834 14358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:08.606986 14358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:08.607049 14358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:08.642113 14358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39271
I20260619 00:56:08.642179 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39271 every 8 connection(s)
I20260619 00:56:08.643555 14358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:56:08.648056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14358
I20260619 00:56:08.648272 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:56:08.649036 388 external_mini_cluster.cc:1428] Setting key 0f7b165b90b70fb5c794a6b16f6156c0
Jun 19 00:56:08 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:08.665854 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:08.646338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58857 (local address 127.0.97.62:37873)
0619 00:56:08.646786 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:08.646791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:08.647849 (+ 1058us) server_negotiation.cc:407] Connection header received
0619 00:56:08.650757 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:08.650762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:08.650829 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:08.650958 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:08.652561 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.653384 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.655014 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.655478 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.655620 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:08.659588 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:08.659594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:08.659599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:08.659640 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:08.662338 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:08.663226 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:08.663233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:08.663241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:08.663376 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:08.663910 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:08.663917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:08.663925 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:08.664224 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:08.664393 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:08.665239 (+ 846us) server_negotiation.cc:299] Negotiation successful
0619 00:56:08.665479 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260619 00:56:08.667068 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:08.646955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:58857)
0619 00:56:08.647676 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:08.647714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:08.649828 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:08.651232 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:08.651242 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:08.651791 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:08.652381 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:08.652398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.653556 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.653561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:08.654855 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:08.654868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:08.655730 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:08.655737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:08.655994 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:08.656723 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:08.656747 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:08.659330 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:08.662507 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:08.662517 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:08.662537 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:08.663088 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:08.663496 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:08.663501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:08.663504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:08.663718 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:08.664364 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:08.664372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:08.664774 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 00:56:08.665136 (+ 362us) client_negotiation.cc:240] Negotiation successful
0619 00:56:08.665747 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260619 00:56:08.668853 14486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:08.669189 14486 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:08.669917 14486 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:56:08.672055 14297 ts_manager.cc:194] Registered new tserver with Master: 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:56:08.673725 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:56:08.700099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:09.045141 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:09.045446 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:09.045625 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:09.053393 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:09.053520 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:09.053696 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:56:09.063906 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:09.065364 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:09.066713 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:09.080385 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:09.080463 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:09.080447 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:09.081252 14492 server_base.cc:1034] running on GCE node
I20260619 00:56:09.081957 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:09.082969 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:09.084230 14492 hybrid_clock.cc:648] HybridClock initialized: now 1781830569084170 us; error 56 us; skew 500 ppm
I20260619 00:56:09.084566 14492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:09 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:09.087833 14492 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:56:09.089408 14492 webserver.cc:460] Webserver started at http://127.0.97.2:46807/ using document root <none> and password file <none>
I20260619 00:56:09.090106 14492 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:09.090260 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:09.090559 14492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:09.093637 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "17b621bd788a4194b1079ccd5f51ea69"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "0a6aea4d4f6b7c94451e62789b66f0ca"
server_key_iv: "57fa141120a59d36ec8f1ced35cd97dc"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.094458 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "17b621bd788a4194b1079ccd5f51ea69"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "0a6aea4d4f6b7c94451e62789b66f0ca"
server_key_iv: "57fa141120a59d36ec8f1ced35cd97dc"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.099851 14492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 00:56:09.103811 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.105497 14492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:56:09.105706 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "17b621bd788a4194b1079ccd5f51ea69"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "0a6aea4d4f6b7c94451e62789b66f0ca"
server_key_iv: "57fa141120a59d36ec8f1ced35cd97dc"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.105930 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:09.119484 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:09.124554 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:09.124963 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:09.125978 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:09.127449 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:09.127523 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.127686 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:09.127753 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.161454 14492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37393
I20260619 00:56:09.161497 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37393 every 8 connection(s)
I20260619 00:56:09.162758 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:56:09.163769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14492
I20260619 00:56:09.163980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:56:09.164489 388 external_mini_cluster.cc:1428] Setting key 2040c067654156be6f344852b14cdae0
Jun 19 00:56:09 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:09.185127 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.165707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53711 (local address 127.0.97.62:37873)
0619 00:56:09.165908 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.165914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.167188 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 00:56:09.169235 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.169241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.169331 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.169449 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:09.170971 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.171680 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.173565 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.174036 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.174158 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.178559 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:09.178564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:09.178567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:09.178642 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:09.181617 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.182484 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.182489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.182492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.182632 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.183221 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.183225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.183227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.183513 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:09.183684 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.184472 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.184737 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:56:09.186185 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.166268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:53711)
0619 00:56:09.167028 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.167071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.168966 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.169688 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.169697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.170240 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:09.170795 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.170812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.171839 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.171843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.173395 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.173411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.174273 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.174282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.174616 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.175396 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:09.175440 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:09.178303 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:09.181807 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:09.181817 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:09.181837 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:09.182326 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:09.182756 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:09.182761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:09.182765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:09.183070 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:09.183666 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:09.183675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:09.184080 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.184363 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.184994 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:56:09.187817 14620 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:09.188176 14620 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:09.189077 14620 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:56:09.190860 14297 ts_manager.cc:194] Registered new tserver with Master: 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393)
I20260619 00:56:09.191748 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:56:09.217016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:09.557747 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:09.558072 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:09.558172 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:09.565788 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:09.565930 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:09.566048 14625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:56:09.576561 14625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:09.577894 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:09.579286 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:09.597186 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:09.597306 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:09.597774 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:09.598538 14625 server_base.cc:1034] running on GCE node
I20260619 00:56:09.599293 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:09.600301 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:09.601593 14625 hybrid_clock.cc:648] HybridClock initialized: now 1781830569601514 us; error 60 us; skew 500 ppm
I20260619 00:56:09.601986 14625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:09 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:09.605559 14625 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:56:09.607245 14625 webserver.cc:460] Webserver started at http://127.0.97.3:41395/ using document root <none> and password file <none>
I20260619 00:56:09.607975 14625 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:09.608125 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:09.608449 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:09.611675 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "8b2f492fd4729e8734542ce2fd99fb11"
server_key_iv: "a89585489745bf379a0e33c47dcfc5f2"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.612530 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "8b2f492fd4729e8734542ce2fd99fb11"
server_key_iv: "a89585489745bf379a0e33c47dcfc5f2"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.617951 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 00:56:09.621628 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.623325 14625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:56:09.623551 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "8b2f492fd4729e8734542ce2fd99fb11"
server_key_iv: "a89585489745bf379a0e33c47dcfc5f2"
server_key_version: "encryptionkey@0"
I20260619 00:56:09.623792 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:09.655972 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:09.660712 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:09.661069 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:09.662122 14625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:09.663578 14625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:09.663688 14625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.663856 14625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:09.663938 14625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:09.676965 14486 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:09.698663 14625 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35153
I20260619 00:56:09.698711 14752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35153 every 8 connection(s)
I20260619 00:56:09.699999 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:56:09.701318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14625
I20260619 00:56:09.701522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:56:09.702131 388 external_mini_cluster.cc:1428] Setting key a1056305fe58b4ad1e7e06c8d7b3d13b
Jun 19 00:56:09 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:09.721778 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.703263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43493 (local address 127.0.97.62:37873)
0619 00:56:09.703833 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.703841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.705180 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 00:56:09.707257 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.707261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.707340 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.707436 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:09.709315 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.710124 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.711729 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.712100 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.712202 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.715686 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:09.715692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:09.715696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:09.715729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:09.718873 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.719682 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.719686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.719690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.719809 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.720199 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.720203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.720205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.720464 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:09.720575 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.721338 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.721509 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260619 00:56:09.722798 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.703988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:43493)
0619 00:56:09.704987 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.705046 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.706974 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.707669 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.707677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.708330 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:09.709107 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.709128 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.710253 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.710256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.711606 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.711618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.712282 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.712287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.712480 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.713214 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:09.713236 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:09.715472 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:09.719032 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:09.719042 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:09.719081 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:09.719570 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:09.719902 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:09.719905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:09.719907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:09.720093 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:09.720571 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:09.720577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:09.720987 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.721220 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.721630 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":394,"threads_started":1}
I20260619 00:56:09.724509 14753 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:09.724980 14753 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:09.725744 14753 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
I20260619 00:56:09.727320 14297 ts_manager.cc:194] Registered new tserver with Master: 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153)
I20260619 00:56:09.728214 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43493
I20260619 00:56:09.731169 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:56:09.749217 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.735576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.97.62:37873)
0619 00:56:09.735749 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.735753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.736248 (+ 495us) server_negotiation.cc:407] Connection header received
0619 00:56:09.736384 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.736387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.736464 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.736549 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:09.737942 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.738798 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.740080 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.740478 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.740569 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.743144 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:09.743150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:09.743154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:09.743187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:09.746263 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.747020 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.747026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.747033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.747191 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:09.747631 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:09.747636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:09.747638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:09.747958 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:09.748215 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.748578 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.748863 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:56:09.753324 14297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:56:09.760254 14297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:56:09.793280 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.772708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.97.1:39271)
0619 00:56:09.783226 (+ 10518us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.783236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.783990 (+ 754us) server_negotiation.cc:407] Connection header received
0619 00:56:09.784382 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.784391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.784669 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.784905 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:09.786556 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.788093 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.790234 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.791681 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.791817 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.792049 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.792722 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.792927 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10327,"thread_start_us":10163,"threads_started":1}
I20260619 00:56:09.793339 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.772060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41650)
0619 00:56:09.783759 (+ 11699us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.783849 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.784082 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.785200 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.785204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.785238 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:09.786320 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.786332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.788272 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.788277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.790071 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.790083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.792187 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.792193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.792401 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.792461 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.792614 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.792834 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11579,"mutex_wait_us":761,"spinlock_wait_cycles":896,"thread_start_us":11112,"threads_started":1}
I20260619 00:56:09.793812 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.772060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37393 (local address 127.0.0.1:57864)
0619 00:56:09.783760 (+ 11700us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.783848 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.784082 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.785190 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.785197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.785244 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:09.785763 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.785779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.787819 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.787824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.790448 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.790464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.792511 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.792517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.792684 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.792711 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.792799 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.792941 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11382,"mutex_wait_us":28,"spinlock_wait_cycles":2816,"thread_start_us":11236,"threads_started":1}
I20260619 00:56:09.793960 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.772136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.97.2:37393)
0619 00:56:09.782791 (+ 10655us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.782804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.784293 (+ 1489us) server_negotiation.cc:407] Connection header received
0619 00:56:09.784509 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.784516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.784709 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.784906 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:09.786003 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.787638 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.790612 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.792048 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.792181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.792364 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.793414 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.793593 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10459,"thread_start_us":10325,"threads_started":1}
I20260619 00:56:09.795090 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.785475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35153 (local address 127.0.0.1:52558)
0619 00:56:09.786841 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.786859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.787040 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.787941 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.787945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.787976 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:09.788507 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.788524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.790557 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.790562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.792070 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.792082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.794306 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.794313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.794507 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.794537 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.794627 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.794758 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":31,"thread_start_us":1088,"threads_started":1}
I20260619 00:56:09.795199 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.784723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.97.3:35153)
0619 00:56:09.785982 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.785991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.786943 (+ 952us) server_negotiation.cc:407] Connection header received
0619 00:56:09.787235 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.787243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.787474 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.787633 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:09.788749 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.790372 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.792268 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.793709 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.793804 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.794016 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.794704 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.794858 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":897,"threads_started":1}
I20260619 00:56:09.797864 14421 tablet_service.cc:1467] Processing CreateTablet for tablet f9fddd49f1634bf4bc3761b8c232032a (DEFAULT_TABLE table=test-table [id=419f35522d164d24806f1ea41681c06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:09.798040 14555 tablet_service.cc:1467] Processing CreateTablet for tablet f9fddd49f1634bf4bc3761b8c232032a (DEFAULT_TABLE table=test-table [id=419f35522d164d24806f1ea41681c06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:09.799047 14421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fddd49f1634bf4bc3761b8c232032a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:09.798602 14688 tablet_service.cc:1467] Processing CreateTablet for tablet f9fddd49f1634bf4bc3761b8c232032a (DEFAULT_TABLE table=test-table [id=419f35522d164d24806f1ea41681c06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:09.799403 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fddd49f1634bf4bc3761b8c232032a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:09.799892 14688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9fddd49f1634bf4bc3761b8c232032a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:09.810851 14772 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Bootstrap starting.
I20260619 00:56:09.814447 14772 tablet_bootstrap.cc:654] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:09.815685 14772 log.cc:824] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:09.815871 14774 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Bootstrap starting.
I20260619 00:56:09.817365 14773 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Bootstrap starting.
I20260619 00:56:09.820281 14774 tablet_bootstrap.cc:654] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:09.821163 14773 tablet_bootstrap.cc:654] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:09.821535 14772 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: No bootstrap required, opened a new log
I20260619 00:56:09.821811 14774 log.cc:824] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:09.821872 14772 ts_tablet_manager.cc:1397] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260619 00:56:09.822304 14773 log.cc:824] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:09.825197 14774 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: No bootstrap required, opened a new log
I20260619 00:56:09.825477 14774 ts_tablet_manager.cc:1397] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260619 00:56:09.825531 14773 tablet_bootstrap.cc:492] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: No bootstrap required, opened a new log
I20260619 00:56:09.825764 14773 ts_tablet_manager.cc:1397] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260619 00:56:09.831103 14772 raft_consensus.cc:348] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.831511 14772 raft_consensus.cc:374] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:09.831637 14772 raft_consensus.cc:729] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc5587acbd748d7ade71413bb09a48b, State: Initialized, Role: FOLLOWER
I20260619 00:56:09.832305 14772 consensus_queue.cc:260] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.834331 14772 ts_tablet_manager.cc:1428] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260619 00:56:09.836321 14774 raft_consensus.cc:348] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.836824 14774 raft_consensus.cc:374] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:09.836927 14774 raft_consensus.cc:729] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b621bd788a4194b1079ccd5f51ea69, State: Initialized, Role: FOLLOWER
I20260619 00:56:09.836650 14773 raft_consensus.cc:348] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.837080 14773 raft_consensus.cc:374] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:09.837173 14773 raft_consensus.cc:729] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959b15f0d7f54d7ba2a3bd8d6225aac8, State: Initialized, Role: FOLLOWER
I20260619 00:56:09.837558 14774 consensus_queue.cc:260] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.837821 14773 consensus_queue.cc:260] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.839056 14753 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:09.840047 14773 ts_tablet_manager.cc:1428] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 00:56:09.844898 14620 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:09.845846 14774 ts_tablet_manager.cc:1428] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Time spent starting tablet: real 0.020s user 0.017s sys 0.002s
W20260619 00:56:09.896286 14487 tablet.cc:2369] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:09.915071 14621 tablet.cc:2369] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:09.942236 14779 raft_consensus.cc:482] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:56:09.942623 14779 raft_consensus.cc:504] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.946132 14779 leader_election.cc:288] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271), 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153)
W20260619 00:56:09.951758 14754 tablet.cc:2369] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:09.951907 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.944998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41005 (local address 127.0.97.1:39271)
0619 00:56:09.945260 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.945269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.945956 (+ 687us) server_negotiation.cc:407] Connection header received
0619 00:56:09.946290 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.946297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.946376 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.946450 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:09.947139 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.947859 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.949961 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.950877 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.951041 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.951158 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.951478 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.951631 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:56:09.951980 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.946020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51049 (local address 127.0.97.3:35153)
0619 00:56:09.946224 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.946229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.946251 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:56:09.946439 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.946444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.946520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.946587 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:09.947315 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.948043 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.950023 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.950914 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.951075 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.951186 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.951495 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.951619 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:56:09.951992 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.945435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35153 (local address 127.0.97.2:51049)
0619 00:56:09.946134 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.946152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.946268 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.946777 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.946780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.946800 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:09.947147 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.947155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.948202 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.948208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.949852 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.949866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.951151 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.951157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.951330 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.951353 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.951462 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.951607 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":933,"threads_started":1}
I20260619 00:56:09.952646 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.944891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.97.2:41005)
0619 00:56:09.945837 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:09.945868 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:09.946080 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:09.946606 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:09.946610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:09.946628 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:09.946975 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.946984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.947996 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.948003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.949781 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:09.949796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.951133 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.951142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:09.951306 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.951330 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:56:09.951436 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 00:56:09.952323 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":576,"threads_started":1}
I20260619 00:56:09.953260 14708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fddd49f1634bf4bc3761b8c232032a" candidate_uuid: "17b621bd788a4194b1079ccd5f51ea69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" is_pre_election: true
I20260619 00:56:09.953794 14708 raft_consensus.cc:2454] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b621bd788a4194b1079ccd5f51ea69 in term 0.
I20260619 00:56:09.953775 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fddd49f1634bf4bc3761b8c232032a" candidate_uuid: "17b621bd788a4194b1079ccd5f51ea69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc5587acbd748d7ade71413bb09a48b" is_pre_election: true
I20260619 00:56:09.954365 14441 raft_consensus.cc:2454] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b621bd788a4194b1079ccd5f51ea69 in term 0.
I20260619 00:56:09.954535 14510 leader_election.cc:302] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b621bd788a4194b1079ccd5f51ea69, 959b15f0d7f54d7ba2a3bd8d6225aac8; no voters:
I20260619 00:56:09.955029 14779 raft_consensus.cc:2781] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:56:09.955180 14779 raft_consensus.cc:482] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:56:09.955286 14779 raft_consensus.cc:3037] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:09.957576 14779 raft_consensus.cc:504] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.958204 14779 leader_election.cc:288] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271), 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153)
I20260619 00:56:09.958720 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fddd49f1634bf4bc3761b8c232032a" candidate_uuid: "17b621bd788a4194b1079ccd5f51ea69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc5587acbd748d7ade71413bb09a48b"
I20260619 00:56:09.958961 14441 raft_consensus.cc:3037] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:09.958887 14708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9fddd49f1634bf4bc3761b8c232032a" candidate_uuid: "17b621bd788a4194b1079ccd5f51ea69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
I20260619 00:56:09.959163 14708 raft_consensus.cc:3037] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:09.961450 14441 raft_consensus.cc:2454] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b621bd788a4194b1079ccd5f51ea69 in term 1.
I20260619 00:56:09.961488 14708 raft_consensus.cc:2454] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b621bd788a4194b1079ccd5f51ea69 in term 1.
I20260619 00:56:09.961961 14511 leader_election.cc:302] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b621bd788a4194b1079ccd5f51ea69, 1cc5587acbd748d7ade71413bb09a48b; no voters:
I20260619 00:56:09.962332 14779 raft_consensus.cc:2781] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:09.962872 14779 raft_consensus.cc:686] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 1 LEADER]: Becoming Leader. State: Replica: 17b621bd788a4194b1079ccd5f51ea69, State: Running, Role: LEADER
I20260619 00:56:09.963390 14779 consensus_queue.cc:237] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } }
I20260619 00:56:09.967991 14296 catalog_manager.cc:5495] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "17b621bd788a4194b1079ccd5f51ea69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } health_report { overall_health: UNKNOWN } } }
I20260619 00:56:09.995899 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:09.990780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.97.2:37393)
0619 00:56:09.990973 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:09.990977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:09.990994 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:56:09.991082 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:09.991086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:09.991159 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:09.991287 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:09.991614 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.992342 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.993772 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:09.994178 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:09.994280 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:09.994775 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:09.995298 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:56:09.995414 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:09.995601 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 00:56:09.995670 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:56:10.005331 14441 raft_consensus.cc:1261] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Refusing update from remote peer 17b621bd788a4194b1079ccd5f51ea69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:10.005373 14708 raft_consensus.cc:1261] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Refusing update from remote peer 17b621bd788a4194b1079ccd5f51ea69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:10.006395 14783 consensus_queue.cc:1035] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:10.006759 14779 consensus_queue.cc:1035] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:10.026932 14788 mvcc.cc:205] Tried to move back new op lower bound from 7298378014730452992 to 7298378014573912064. Current Snapshot: MvccSnapshot[applied={T|T < 7298378014730452992}]
I20260619 00:56:10.040593 14296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:10.040938 14296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:10.044942 14296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Sending DeleteTablet for 3 replicas of tablet f9fddd49f1634bf4bc3761b8c232032a
I20260619 00:56:10.046221 14421 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fddd49f1634bf4bc3761b8c232032a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41650
I20260619 00:56:10.046226 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fddd49f1634bf4bc3761b8c232032a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57864
I20260619 00:56:10.046447 14688 tablet_service.cc:1514] Processing DeleteTablet for tablet f9fddd49f1634bf4bc3761b8c232032a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52558
I20260619 00:56:10.047855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14358
I20260619 00:56:10.061667 14796 tablet_replica.cc:330] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: stopping tablet replica
I20260619 00:56:10.062494 14796 raft_consensus.cc:2229] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:56:10.063468 14796 raft_consensus.cc:2258] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:10.066233 14796 ts_tablet_manager.cc:1905] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:10.074946 14798 tablet_replica.cc:330] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: stopping tablet replica
I20260619 00:56:10.075033 14796 ts_tablet_manager.cc:1918] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:10.075291 14796 log.cc:1196] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/f9fddd49f1634bf4bc3761b8c232032a
I20260619 00:56:10.075572 14798 raft_consensus.cc:2229] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:10.076076 14796 ts_tablet_manager.cc:1939] T f9fddd49f1634bf4bc3761b8c232032a P 17b621bd788a4194b1079ccd5f51ea69: Deleting consensus metadata
I20260619 00:56:10.076587 14798 raft_consensus.cc:2258] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:10.078166 14283 catalog_manager.cc:4841] TS 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393): tablet f9fddd49f1634bf4bc3761b8c232032a (table test-table [id=419f35522d164d24806f1ea41681c06b]) successfully deleted
I20260619 00:56:10.082791 14798 ts_tablet_manager.cc:1905] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:10.083456 14797 tablet_replica.cc:330] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: stopping tablet replica
I20260619 00:56:10.083988 14797 raft_consensus.cc:2229] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:10.084565 14797 raft_consensus.cc:2258] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 00:56:10 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:56:10.090193 14797 ts_tablet_manager.cc:1905] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:10.092157 14798 ts_tablet_manager.cc:1918] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:10.092415 14798 log.cc:1196] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/f9fddd49f1634bf4bc3761b8c232032a
I20260619 00:56:10.093101 14798 ts_tablet_manager.cc:1939] T f9fddd49f1634bf4bc3761b8c232032a P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting consensus metadata
I20260619 00:56:10.094954 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.049447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.1:39271)
0619 00:56:10.049678 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:10.049687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:10.071622 (+ 21935us) server_negotiation.cc:407] Connection header received
0619 00:56:10.072172 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:10.072180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:10.072305 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:10.072381 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:10.076861 (+ 4480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.078013 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.080407 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.081144 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.081286 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:10.086985 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:10.087001 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:10.087021 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:10.087071 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:10.091106 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.092022 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.092030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.092034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.092226 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.092776 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.092782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.092785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.093326 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:10.093616 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:10.094288 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 00:56:10.094533 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:56:10.095230 14283 catalog_manager.cc:4841] TS 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153): tablet f9fddd49f1634bf4bc3761b8c232032a (table test-table [id=419f35522d164d24806f1ea41681c06b]) successfully deleted
I20260619 00:56:10.096684 14481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:10.200579 14797 ts_tablet_manager.cc:1918] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:10.200896 14797 log.cc:1196] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/f9fddd49f1634bf4bc3761b8c232032a
I20260619 00:56:10.201251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14358
I20260619 00:56:10.219256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14492
W20260619 00:56:10.219240 14284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39271 (error 108)
I20260619 00:56:10.221428 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.220849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41680)
0619 00:56:10.221116 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.221241 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260619 00:56:10.221683 14284 catalog_manager.cc:4584] TS 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9fddd49f1634bf4bc3761b8c232032a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Jun 19 00:56:10 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:56:10.242551 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.220137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.0.97.2:37393)
0619 00:56:10.220342 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:10.220349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:10.220620 (+ 271us) server_negotiation.cc:407] Connection header received
0619 00:56:10.220685 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:10.220689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:10.220755 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:10.220912 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:10.222475 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.223736 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.225253 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.225938 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.226070 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:10.235374 (+ 9304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:10.235393 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:10.235414 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:10.235473 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:10.239572 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.240301 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.240311 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.240315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.240511 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.240889 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.240894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.240897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.241389 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:10.241692 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:10.241885 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 00:56:10.242118 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:56:10.243427 14615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:10.275687 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.275108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41696)
0619 00:56:10.275368 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.275490 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 00:56:10.276892 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.276436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41710)
0619 00:56:10.276606 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.276702 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 00:56:10.337525 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.336980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41716)
0619 00:56:10.337219 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.337340 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 00:56:10.338675 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.338271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41722)
0619 00:56:10.338447 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.338533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 00:56:10.361615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14492
I20260619 00:56:10.381232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14625
Jun 19 00:56:10 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:56:10.397496 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.382099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.97.3:35153)
0619 00:56:10.382285 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:10.382290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:10.382369 (+ 79us) server_negotiation.cc:407] Connection header received
0619 00:56:10.382532 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:10.382535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:10.382600 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:10.382685 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:10.383871 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.384595 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.386023 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:10.386450 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:10.386558 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:10.391084 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:10.391100 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:10.391118 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:10.391163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:10.394187 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.395052 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.395061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.395064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.395223 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:10.395767 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:10.395772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:10.395774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:10.396322 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:10.396565 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:10.396970 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 00:56:10.397155 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:56:10.398281 14748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:10.419610 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.419064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41732)
0619 00:56:10.419336 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.419437 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 00:56:10.420876 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:10.420402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:41742)
0619 00:56:10.420607 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:10.420689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 00:56:10.490319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14625
I20260619 00:56:10.507540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14265
I20260619 00:56:10.508863 14327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:10.636991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14265
I20260619 00:56:10.659664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--webserver_interface=127.0.97.62
--webserver_port=43571
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:11.008683 14805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:11.009168 14805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:11.009334 14805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:11.017587 14805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:11.017781 14805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:11.017907 14805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:11.018013 14805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:11.018097 14805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:11.028685 14805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:11.030176 14805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:11.031672 14805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:11.055146 14811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:11.055151 14812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:11.055176 14814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:11.056481 14805 server_base.cc:1034] running on GCE node
I20260619 00:56:11.057576 14805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:11.059391 14805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:11.060613 14805 hybrid_clock.cc:648] HybridClock initialized: now 1781830571060539 us; error 73 us; skew 500 ppm
I20260619 00:56:11.061115 14805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:11 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:11.064999 14805 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:11.066860 14805 webserver.cc:460] Webserver started at http://127.0.97.62:43571/ using document root <none> and password file <none>
I20260619 00:56:11.067698 14805 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:11.067871 14805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:11.074373 14805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:56:11.077790 14821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:11.079381 14805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:56:11.079608 14805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "ab53d0096d0c408797a3fb950e03e5fb"
format_stamp: "Formatted at 2026-06-19 00:56:07 on dist-test-slave-r7bh"
server_key: "49f5923af0afb07f928e0a812199e4b8"
server_key_iv: "1f798e46a33143907c4301cecd8b0254"
server_key_version: "encryptionkey@0"
I20260619 00:56:11.080777 14805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:11.110307 14805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:11.115257 14805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:11.115568 14805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:11.135763 14805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37873
I20260619 00:56:11.135809 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37873 every 8 connection(s)
I20260619 00:56:11.137472 14805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:11.144177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14805
I20260619 00:56:11.145262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=42005
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:11.145576 14873 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:56:11.151293 14873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap starting.
I20260619 00:56:11.172428 14873 log.cc:824] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:11.186537 14873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:56:11.187141 14873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap complete.
I20260619 00:56:11.196576 14873 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:11.197549 14873 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Initialized, Role: FOLLOWER
I20260619 00:56:11.198338 14873 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:11.198590 14873 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:11.198741 14873 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:11.198932 14873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:56:11.203122 14873 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:11.203740 14873 leader_election.cc:302] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab53d0096d0c408797a3fb950e03e5fb; no voters:
I20260619 00:56:11.204465 14873 leader_election.cc:288] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 00:56:11.204607 14877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Leader election won for term 2
I20260619 00:56:11.205796 14877 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 LEADER]: Becoming Leader. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Running, Role: LEADER
I20260619 00:56:11.206458 14877 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:11.206910 14873 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:11.209347 14878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:11.209653 14878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:11.208973 14879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab53d0096d0c408797a3fb950e03e5fb. Latest consensus state: current_term: 2 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:11.210228 14879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:11.213986 14883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:11.224664 14883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:11.226795 14883 catalog_manager.cc:1259] Loaded cluster ID: 592d6140578a4be9b73a659b607c7ee5
I20260619 00:56:11.227370 14883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:11.231900 14883 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 00:56:11.249799 14883 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 00:56:11.257334 14883 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:11.259063 14883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Loaded TSK: 0
I20260619 00:56:11.260387 14883 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 00:56:11.529723 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:11.530112 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:11.530282 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:11.538369 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:11.538553 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:11.538738 14875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:56:11.549870 14875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42005
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:11.551427 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:11.552989 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:11.569890 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:11.569929 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:11.570724 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:11.571051 14875 server_base.cc:1034] running on GCE node
I20260619 00:56:11.571846 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:11.573056 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:11.574299 14875 hybrid_clock.cc:648] HybridClock initialized: now 1781830571574229 us; error 76 us; skew 500 ppm
I20260619 00:56:11.574676 14875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:11 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:11.578397 14875 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:56:11.580271 14875 webserver.cc:460] Webserver started at http://127.0.97.1:42005/ using document root <none> and password file <none>
I20260619 00:56:11.581171 14875 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:11.581348 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:11.587926 14875 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 00:56:11.591329 14910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:11.593144 14875 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:56:11.593362 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "1cc5587acbd748d7ade71413bb09a48b"
format_stamp: "Formatted at 2026-06-19 00:56:08 on dist-test-slave-r7bh"
server_key: "25513c71ba9d259fedbe8c9b454b7cea"
server_key_iv: "25b5235e0db9b6db754ceb7b2d02be4c"
server_key_version: "encryptionkey@0"
I20260619 00:56:11.594455 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:11.620560 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:11.625841 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:11.626168 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:11.627358 14875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:11.629395 14917 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 00:56:11.635857 14917 ts_tablet_manager.cc:1852] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 00:56:11.637146 14917 ts_tablet_manager.cc:1905] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:11.646472 14917 ts_tablet_manager.cc:1918] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:11.646885 14917 ts_tablet_manager.cc:1939] T f9fddd49f1634bf4bc3761b8c232032a P 1cc5587acbd748d7ade71413bb09a48b: Deleting consensus metadata
I20260619 00:56:11.647615 14875 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 00:56:11.647774 14875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.000s sys 0.000s
I20260619 00:56:11.647977 14875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:11.648087 14875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:11.686658 14875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39271
I20260619 00:56:11.686692 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39271 every 8 connection(s)
I20260619 00:56:11.688158 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:56:11.693301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 14875
I20260619 00:56:11.694129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37393
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=46807
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:56:11 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:11.717504 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:11.691353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44873 (local address 127.0.97.62:37873)
0619 00:56:11.692050 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:11.692073 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:11.692633 (+ 560us) server_negotiation.cc:407] Connection header received
0619 00:56:11.696146 (+ 3513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:11.696161 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:11.696590 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:11.697059 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:11.699371 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:11.700874 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:11.702590 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:11.703275 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:11.703509 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:11.707784 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:11.707793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:11.707811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:11.707855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:11.711576 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:11.712574 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:11.712585 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:11.712595 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:11.712863 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:11.713596 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:11.713604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:11.713609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:11.714352 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:11.714506 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:11.715841 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0619 00:56:11.716312 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260619 00:56:11.717938 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:11.691574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:44873)
0619 00:56:11.692443 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:11.692483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:11.695402 (+ 2919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:11.697239 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:11.697254 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:11.698010 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:11.699082 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:11.699107 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:11.701077 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:11.701082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:11.702419 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:11.702433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:11.703540 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:11.703548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:11.703868 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:11.704647 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:11.704672 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:11.707440 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:11.711788 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:11.711802 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:11.711822 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:11.712352 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:11.713062 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:11.713070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:11.713076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:11.713393 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:11.714526 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:11.714539 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:11.715180 (+ 641us) client_negotiation.cc:769] Sending connection context
0619 00:56:11.715626 (+ 446us) client_negotiation.cc:240] Negotiation successful
0619 00:56:11.716341 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":342,"threads_started":1}
I20260619 00:56:11.721119 15024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:11.721489 15024 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:11.722411 15024 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
I20260619 00:56:11.725147 14838 ts_manager.cc:194] Registered new tserver with Master: 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271)
I20260619 00:56:11.726708 14838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44873
W20260619 00:56:12.066828 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:12.067162 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:12.067340 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:12.074968 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:12.075114 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:12.075315 15028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:56:12.085700 15028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46807
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:12.087113 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:12.088488 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:12.102916 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:12.102977 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:12.103446 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:12.103866 15028 server_base.cc:1034] running on GCE node
I20260619 00:56:12.104561 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:12.105720 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:12.106971 15028 hybrid_clock.cc:648] HybridClock initialized: now 1781830572106921 us; error 42 us; skew 500 ppm
I20260619 00:56:12.107369 15028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:12 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:12.111011 15028 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:56:12.112702 15028 webserver.cc:460] Webserver started at http://127.0.97.2:46807/ using document root <none> and password file <none>
I20260619 00:56:12.113511 15028 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:12.113643 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:12.120362 15028 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:56:12.123615 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:12.125250 15028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:56:12.125458 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "17b621bd788a4194b1079ccd5f51ea69"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "0a6aea4d4f6b7c94451e62789b66f0ca"
server_key_iv: "57fa141120a59d36ec8f1ced35cd97dc"
server_key_version: "encryptionkey@0"
I20260619 00:56:12.126534 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:12.141804 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:12.146466 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:12.146759 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:12.147900 15028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:12.149336 15028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:12.149468 15028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:12.149598 15028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:12.149684 15028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:12.183130 15028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37393
I20260619 00:56:12.183245 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37393 every 8 connection(s)
I20260619 00:56:12.184738 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:56:12.190882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15028
I20260619 00:56:12.191512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35153
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=41395
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:56:12 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:12.207299 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.187632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42047 (local address 127.0.97.62:37873)
0619 00:56:12.187857 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.187863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.189108 (+ 1245us) server_negotiation.cc:407] Connection header received
0619 00:56:12.190970 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.190976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.191104 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.191229 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:12.192898 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.193632 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.195227 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.195653 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.195793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.199718 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:12.199728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:12.199736 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:12.199785 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:12.203593 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.204447 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.204456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.204463 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.204634 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.205218 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.205226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.205233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.205627 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:12.205782 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.206522 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.206825 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:56:12.208134 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.188106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:42047)
0619 00:56:12.188934 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.188982 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.190656 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.191482 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.191491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.192118 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:12.192705 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.192723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.193802 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.193814 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.195062 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.195073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.195888 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.195895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.196135 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.196931 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:12.196954 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:12.199449 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:12.203780 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:12.203788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:12.203802 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:12.204296 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:12.204783 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:12.204787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:12.204790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:12.205064 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:12.205756 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:12.205763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:12.206109 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.206354 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.206855 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":284,"threads_started":1}
I20260619 00:56:12.209899 15156 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:12.210238 15156 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:12.211014 15156 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
I20260619 00:56:12.212554 14838 ts_manager.cc:194] Registered new tserver with Master: 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393)
I20260619 00:56:12.213500 14838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42047
W20260619 00:56:12.534409 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:12.534706 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:12.534878 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:12.542519 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:12.542657 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:12.542867 15159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:56:12.553134 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41395
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:12.554520 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:12.556034 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:12.570922 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:12.570963 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:12.570914 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:12.572260 15159 server_base.cc:1034] running on GCE node
I20260619 00:56:12.573055 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:12.574086 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:12.575356 15159 hybrid_clock.cc:648] HybridClock initialized: now 1781830572575291 us; error 52 us; skew 500 ppm
I20260619 00:56:12.575757 15159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:12 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:12.579737 15159 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:56:12.581671 15159 webserver.cc:460] Webserver started at http://127.0.97.3:41395/ using document root <none> and password file <none>
I20260619 00:56:12.582472 15159 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:12.582649 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:12.589655 15159 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:56:12.593182 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:12.594821 15159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:56:12.595041 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "8b2f492fd4729e8734542ce2fd99fb11"
server_key_iv: "a89585489745bf379a0e33c47dcfc5f2"
server_key_version: "encryptionkey@0"
I20260619 00:56:12.596134 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:12.636668 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:12.641601 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:12.641916 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:12.643038 15159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:12.644552 15159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:12.644692 15159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:12.644862 15159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:12.644969 15159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 00:56:12.680038 15159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35153
I20260619 00:56:12.680160 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35153 every 8 connection(s)
I20260619 00:56:12.681784 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:56:12.689236 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15159
Jun 19 00:56:12 dist-test-slave-r7bh krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:12.707579 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.684728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39253 (local address 127.0.97.62:37873)
0619 00:56:12.684984 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.684991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.686653 (+ 1662us) server_negotiation.cc:407] Connection header received
0619 00:56:12.689536 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.689544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.689648 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.689779 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:12.692130 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.693334 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.695543 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.696125 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.696307 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.700672 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:12.700680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:12.700685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:12.700727 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:12.703807 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.704723 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.704731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.704735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.704936 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.705442 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.705448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.705451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.705800 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:12.705983 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.706860 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.707107 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 00:56:12.708817 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.685371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:39253)
0619 00:56:12.686422 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.686487 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.689167 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.690074 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.690088 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.690916 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:12.691871 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.691910 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.693600 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.693607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.695392 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.695405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.696409 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.696417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.696707 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.697506 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:12.697532 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:12.700409 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:12.704029 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:12.704038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:12.704058 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:12.704586 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:12.705074 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:12.705077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:12.705080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:12.705283 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:12.705964 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:12.705973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:12.706382 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.706694 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.707406 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":392,"threads_started":1}
I20260619 00:56:12.709930 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.690349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.97.62:37873)
0619 00:56:12.694033 (+ 3684us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.694039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.694458 (+ 419us) server_negotiation.cc:407] Connection header received
0619 00:56:12.694595 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.694612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.694690 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.694819 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:12.696341 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.697348 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.699127 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.699674 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.700079 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.703456 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:12.703466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:12.703472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:12.703546 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:12.706708 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.707602 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.707608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.707612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.707817 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.708360 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.708365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.708368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.708654 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:12.708984 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.709298 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.709545 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3548,"thread_start_us":3360,"threads_started":1}
I20260619 00:56:12.711316 15287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37873
I20260619 00:56:12.711696 15287 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:12.712735 15287 heartbeater.cc:511] Master 127.0.97.62:37873 requested a full tablet report, sending...
I20260619 00:56:12.714455 14837 ts_manager.cc:194] Registered new tserver with Master: 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153)
I20260619 00:56:12.715437 14837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39253
I20260619 00:56:12.726621 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:56:12.730676 15024 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:12.734687 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.727642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.97.62:37873)
0619 00:56:12.727893 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.727901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.728179 (+ 278us) server_negotiation.cc:407] Connection header received
0619 00:56:12.728380 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.728384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.728529 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.728670 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:12.729247 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.730023 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.731923 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.732603 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.732719 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.733373 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:12.733963 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:56:12.734145 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.734301 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.734379 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:56:12.735831 14837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:56:12.743376 14837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:56:12.771719 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.757154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.0.97.2:37393)
0619 00:56:12.757833 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.757847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.758657 (+ 810us) server_negotiation.cc:407] Connection header received
0619 00:56:12.759069 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.759077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.759355 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.759526 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:12.760723 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.762689 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.767531 (+ 4842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.769617 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.769765 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.770045 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.770967 (+ 922us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.771279 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":293,"threads_started":1}
I20260619 00:56:12.772827 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.756931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37393 (local address 127.0.0.1:41590)
0619 00:56:12.758498 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.758538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.758812 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.759864 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.759874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.759923 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:12.760467 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.760483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.762968 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.762976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.767235 (+ 4259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.767264 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.770314 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.770324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.770632 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.770705 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.772060 (+ 1355us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.772323 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"thread_start_us":1175,"threads_started":1}
I20260619 00:56:12.780020 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.757757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.0.1:54366)
0619 00:56:12.770423 (+ 12666us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.770451 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.770606 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.772218 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.772224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.772255 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:12.772686 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.772700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.774864 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.774871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.777215 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.777234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.779187 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.779193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.779349 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.779387 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.779492 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.779660 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12470,"thread_start_us":14532,"threads_started":1}
I20260619 00:56:12.780083 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.758146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.0.97.1:39271)
0619 00:56:12.771163 (+ 13017us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.771173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.771206 (+ 33us) server_negotiation.cc:407] Connection header received
0619 00:56:12.771314 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.771324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.771630 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.772877 (+ 1247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:12.773110 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.774648 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.777444 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.778951 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.779105 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.779351 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.779567 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.779738 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12820,"thread_start_us":13782,"threads_started":1}
I20260619 00:56:12.781781 15091 tablet_service.cc:1467] Processing CreateTablet for tablet e99d734d0bb341e298f53815c5942bae (DEFAULT_TABLE table=test-table [id=3988dee9106d40c0bcf5ba3a12480c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:12.782680 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.759763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35153 (local address 127.0.0.1:45882)
0619 00:56:12.773192 (+ 13429us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.773218 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.773354 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.774565 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.774573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.774625 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:12.774854 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.774868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.776937 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.776942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.778685 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.778698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.781863 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.781869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.782062 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.782100 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.782214 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.782374 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13294,"thread_start_us":16104,"threads_started":1}
I20260619 00:56:12.782827 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.759333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.97.3:35153)
0619 00:56:12.773035 (+ 13702us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.773044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.773496 (+ 452us) server_negotiation.cc:407] Connection header received
0619 00:56:12.773662 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.773673 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.773990 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.774222 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:12.775084 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.776737 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.778869 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.781669 (+ 2800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.781836 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.782082 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.782274 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.782445 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13529,"thread_start_us":13344,"threads_started":1}
I20260619 00:56:12.782454 14959 tablet_service.cc:1467] Processing CreateTablet for tablet e99d734d0bb341e298f53815c5942bae (DEFAULT_TABLE table=test-table [id=3988dee9106d40c0bcf5ba3a12480c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:12.783840 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99d734d0bb341e298f53815c5942bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:12.783859 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99d734d0bb341e298f53815c5942bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:12.787374 15222 tablet_service.cc:1467] Processing CreateTablet for tablet e99d734d0bb341e298f53815c5942bae (DEFAULT_TABLE table=test-table [id=3988dee9106d40c0bcf5ba3a12480c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:12.788970 15222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99d734d0bb341e298f53815c5942bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:12.795831 15303 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Bootstrap starting.
I20260619 00:56:12.798345 15304 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Bootstrap starting.
I20260619 00:56:12.800668 15303 tablet_bootstrap.cc:654] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:12.802186 15303 log.cc:824] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:12.802498 15304 tablet_bootstrap.cc:654] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:12.803932 15304 log.cc:824] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:12.805573 15303 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: No bootstrap required, opened a new log
I20260619 00:56:12.805862 15303 ts_tablet_manager.cc:1397] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 00:56:12.807080 15307 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Bootstrap starting.
I20260619 00:56:12.807430 15304 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: No bootstrap required, opened a new log
I20260619 00:56:12.807740 15304 ts_tablet_manager.cc:1397] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260619 00:56:12.811442 15307 tablet_bootstrap.cc:654] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:12.812760 15307 log.cc:824] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:12.815744 15304 raft_consensus.cc:348] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.816146 15304 raft_consensus.cc:374] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:12.816287 15307 tablet_bootstrap.cc:492] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: No bootstrap required, opened a new log
I20260619 00:56:12.816310 15304 raft_consensus.cc:729] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b621bd788a4194b1079ccd5f51ea69, State: Initialized, Role: FOLLOWER
I20260619 00:56:12.816588 15307 ts_tablet_manager.cc:1397] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 00:56:12.817045 15304 consensus_queue.cc:260] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.817238 15303 raft_consensus.cc:348] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.817723 15303 raft_consensus.cc:374] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:12.817840 15303 raft_consensus.cc:729] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc5587acbd748d7ade71413bb09a48b, State: Initialized, Role: FOLLOWER
I20260619 00:56:12.818588 15303 consensus_queue.cc:260] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.820540 15303 ts_tablet_manager.cc:1428] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 00:56:12.821637 15156 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:12.822312 15304 ts_tablet_manager.cc:1428] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260619 00:56:12.829232 15307 raft_consensus.cc:348] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.829718 15307 raft_consensus.cc:374] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:12.829830 15307 raft_consensus.cc:729] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959b15f0d7f54d7ba2a3bd8d6225aac8, State: Initialized, Role: FOLLOWER
I20260619 00:56:12.830427 15307 consensus_queue.cc:260] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.831632 15287 heartbeater.cc:503] Master 127.0.97.62:37873 was elected leader, sending a full tablet report...
I20260619 00:56:12.832175 15307 ts_tablet_manager.cc:1428] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 00:56:12.849943 15311 raft_consensus.cc:482] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:56:12.850359 15311 raft_consensus.cc:504] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.852227 15311 leader_election.cc:288] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393), 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271)
I20260619 00:56:12.858258 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.853175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57495 (local address 127.0.97.2:37393)
0619 00:56:12.853361 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.853365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.853382 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:56:12.853444 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.853447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.853502 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.853613 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:12.854020 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.854725 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.856358 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.857266 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.857446 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.857542 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.857861 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.857992 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:56:12.858282 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.852412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37393 (local address 127.0.97.3:57495)
0619 00:56:12.852786 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.852834 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.852985 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.853640 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.853645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.853676 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:12.853870 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.853878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.854870 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.854876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.856190 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.856202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.857480 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.857486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.857679 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.857717 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.857829 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.857986 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260619 00:56:12.859398 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.852741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46007 (local address 127.0.97.1:39271)
0619 00:56:12.853029 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.853037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.853745 (+ 708us) server_negotiation.cc:407] Connection header received
0619 00:56:12.853959 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.853963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.854023 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.854121 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:12.854647 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.855354 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.857134 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.858411 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.858548 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.858655 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.859006 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.859142 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 00:56:12.859441 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.852734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39271 (local address 127.0.97.3:46007)
0619 00:56:12.853609 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:12.853634 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:12.853808 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:12.854147 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:12.854151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:12.854175 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:12.854484 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.854491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.855508 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.855518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.856971 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:12.856983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.858611 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.858617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:12.858816 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.858858 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 00:56:12.858972 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 00:56:12.859105 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":500,"threads_started":1}
I20260619 00:56:12.859634 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99d734d0bb341e298f53815c5942bae" candidate_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b621bd788a4194b1079ccd5f51ea69" is_pre_election: true
I20260619 00:56:12.860250 15111 raft_consensus.cc:2454] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 959b15f0d7f54d7ba2a3bd8d6225aac8 in term 0.
I20260619 00:56:12.860481 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99d734d0bb341e298f53815c5942bae" candidate_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc5587acbd748d7ade71413bb09a48b" is_pre_election: true
I20260619 00:56:12.861069 14979 raft_consensus.cc:2454] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 959b15f0d7f54d7ba2a3bd8d6225aac8 in term 0.
I20260619 00:56:12.861032 15177 leader_election.cc:302] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b621bd788a4194b1079ccd5f51ea69, 959b15f0d7f54d7ba2a3bd8d6225aac8; no voters:
I20260619 00:56:12.861521 15311 raft_consensus.cc:2781] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:56:12.861711 15311 raft_consensus.cc:482] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:56:12.861843 15311 raft_consensus.cc:3037] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:12.864207 15311 raft_consensus.cc:504] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.864948 15311 leader_election.cc:288] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [CANDIDATE]: Term 1 election: Requested vote from peers 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393), 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271)
I20260619 00:56:12.865551 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99d734d0bb341e298f53815c5942bae" candidate_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b621bd788a4194b1079ccd5f51ea69"
I20260619 00:56:12.865550 14979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99d734d0bb341e298f53815c5942bae" candidate_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc5587acbd748d7ade71413bb09a48b"
I20260619 00:56:12.865790 15111 raft_consensus.cc:3037] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:12.865794 14979 raft_consensus.cc:3037] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:12.869428 15111 raft_consensus.cc:2454] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 959b15f0d7f54d7ba2a3bd8d6225aac8 in term 1.
I20260619 00:56:12.869428 14979 raft_consensus.cc:2454] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 959b15f0d7f54d7ba2a3bd8d6225aac8 in term 1.
I20260619 00:56:12.870118 15178 leader_election.cc:302] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc5587acbd748d7ade71413bb09a48b, 959b15f0d7f54d7ba2a3bd8d6225aac8; no voters:
I20260619 00:56:12.870515 15311 raft_consensus.cc:2781] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:12.871032 15311 raft_consensus.cc:686] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 LEADER]: Becoming Leader. State: Replica: 959b15f0d7f54d7ba2a3bd8d6225aac8, State: Running, Role: LEADER
I20260619 00:56:12.871654 15311 consensus_queue.cc:237] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } }
I20260619 00:56:12.877012 14837 catalog_manager.cc:5495] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 } health_report { overall_health: UNKNOWN } } }
I20260619 00:56:12.916641 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.910920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.97.3:35153)
0619 00:56:12.911136 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.911140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.911160 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:56:12.911325 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.911329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.911411 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.911549 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:12.911913 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.912624 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.914281 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.914754 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.914864 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.915386 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:12.915981 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:56:12.916102 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.916208 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.916344 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:56:12.925659 15111 raft_consensus.cc:1261] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Refusing update from remote peer 959b15f0d7f54d7ba2a3bd8d6225aac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:12.925659 14979 raft_consensus.cc:1261] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Refusing update from remote peer 959b15f0d7f54d7ba2a3bd8d6225aac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:12.926892 15313 consensus_queue.cc:1035] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc5587acbd748d7ade71413bb09a48b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:12.927291 15311 consensus_queue.cc:1035] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b621bd788a4194b1079ccd5f51ea69" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:56:12.933737 15288 tablet.cc:2369] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:12.937036 15157 tablet.cc:2369] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:12.941067 15025 tablet.cc:2369] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:12.941946 15316 mvcc.cc:205] Tried to move back new op lower bound from 7298378026691710976 to 7298378026486452224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378026691710976}]
I20260619 00:56:12.944149 15317 mvcc.cc:205] Tried to move back new op lower bound from 7298378026691710976 to 7298378026486452224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378026691710976}]
I20260619 00:56:12.946076 15318 mvcc.cc:205] Tried to move back new op lower bound from 7298378026691710976 to 7298378026486452224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378026691710976}]
I20260619 00:56:12.959975 14838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:12.960218 14838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:12.964625 14838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Sending DeleteTablet for 3 replicas of tablet e99d734d0bb341e298f53815c5942bae
I20260619 00:56:12.966167 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet e99d734d0bb341e298f53815c5942bae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41590
I20260619 00:56:12.966234 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet e99d734d0bb341e298f53815c5942bae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260619 00:56:12.966346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14875
I20260619 00:56:12.967345 15325 tablet_replica.cc:330] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: stopping tablet replica
I20260619 00:56:12.967839 15325 raft_consensus.cc:2229] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:12.967792 15222 tablet_service.cc:1514] Processing DeleteTablet for tablet e99d734d0bb341e298f53815c5942bae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45882
I20260619 00:56:12.968420 15325 raft_consensus.cc:2258] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:12.968781 15326 tablet_replica.cc:330] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: stopping tablet replica
I20260619 00:56:12.971009 15325 ts_tablet_manager.cc:1905] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:12.976611 15326 raft_consensus.cc:2229] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:56:12.977543 15326 raft_consensus.cc:2258] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:12.978799 15327 tablet_replica.cc:330] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: stopping tablet replica
I20260619 00:56:12.979206 15325 ts_tablet_manager.cc:1918] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:12.979478 15325 log.cc:1196] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/e99d734d0bb341e298f53815c5942bae
I20260619 00:56:12.980144 15325 ts_tablet_manager.cc:1939] T e99d734d0bb341e298f53815c5942bae P 1cc5587acbd748d7ade71413bb09a48b: Deleting consensus metadata
I20260619 00:56:12.980201 15326 ts_tablet_manager.cc:1905] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:12.982354 14825 catalog_manager.cc:4841] TS 1cc5587acbd748d7ade71413bb09a48b (127.0.97.1:39271): tablet e99d734d0bb341e298f53815c5942bae (table test-table [id=3988dee9106d40c0bcf5ba3a12480c95]) successfully deleted
I20260619 00:56:12.983350 15327 raft_consensus.cc:2229] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:12.984010 15327 raft_consensus.cc:2258] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:12.986811 15327 ts_tablet_manager.cc:1905] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:12.988354 15326 ts_tablet_manager.cc:1918] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:12.988606 15326 log.cc:1196] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/e99d734d0bb341e298f53815c5942bae
I20260619 00:56:12.989295 15326 ts_tablet_manager.cc:1939] T e99d734d0bb341e298f53815c5942bae P 959b15f0d7f54d7ba2a3bd8d6225aac8: Deleting consensus metadata
I20260619 00:56:12.991273 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:12.968531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.0.97.1:39271)
0619 00:56:12.968727 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:12.968734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:12.968755 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:56:12.968854 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:12.968860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:12.968956 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:12.969086 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:12.970979 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.972079 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.974132 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:12.974814 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:12.974958 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:12.982132 (+ 7174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:12.982148 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:12.982166 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:12.982210 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:12.986386 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.987505 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.987518 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.987524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.987739 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:12.988218 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:12.988223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:12.988227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:12.988773 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:12.989162 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:12.990607 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0619 00:56:12.990879 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:56:12.991410 14824 catalog_manager.cc:4841] TS 959b15f0d7f54d7ba2a3bd8d6225aac8 (127.0.97.3:35153): tablet e99d734d0bb341e298f53815c5942bae (table test-table [id=3988dee9106d40c0bcf5ba3a12480c95]) successfully deleted
I20260619 00:56:12.992395 15019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:12.994673 15327 ts_tablet_manager.cc:1918] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:12.994899 15327 log.cc:1196] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/e99d734d0bb341e298f53815c5942bae
I20260619 00:56:12.995497 15327 ts_tablet_manager.cc:1939] T e99d734d0bb341e298f53815c5942bae P 17b621bd788a4194b1079ccd5f51ea69: Deleting consensus metadata
I20260619 00:56:12.997575 14824 catalog_manager.cc:4841] TS 17b621bd788a4194b1079ccd5f51ea69 (127.0.97.2:37393): tablet e99d734d0bb341e298f53815c5942bae (table test-table [id=3988dee9106d40c0bcf5ba3a12480c95]) successfully deleted
I20260619 00:56:13.087934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14875
I20260619 00:56:13.107178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15028
I20260619 00:56:13.125856 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:13.108171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.0.97.2:37393)
0619 00:56:13.108400 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:13.108407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:13.108432 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:56:13.108508 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:13.108513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:13.108597 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:13.108717 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:13.110183 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:13.111403 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:13.113291 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:13.114057 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:13.114284 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:13.117951 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:13.117976 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:13.118000 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:13.118061 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:13.122321 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:13.123279 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:13.123291 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:13.123294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:13.123467 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:13.123943 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:13.123948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:13.123952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:13.124466 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:13.124777 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:13.125204 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 00:56:13.125456 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:56:13.126755 15151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:13.229578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15028
I20260619 00:56:13.253611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15159
I20260619 00:56:13.269227 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:13.254528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.97.3:35153)
0619 00:56:13.254766 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:13.254772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:13.254794 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:56:13.254874 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:13.254878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:13.254953 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:13.255065 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:13.256437 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:13.257363 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:13.258806 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:13.259284 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:13.259445 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:13.262259 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:13.262272 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:13.262292 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:13.262346 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:13.265929 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:13.266838 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:13.266847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:13.266849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:13.267015 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:13.267444 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:13.267449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:13.267454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:13.267852 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:13.268211 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:13.268516 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 00:56:13.268787 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:56:13.270092 15282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:13.398170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15159
I20260619 00:56:13.419775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 14805
I20260619 00:56:13.421214 14868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:13.538280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 14805
I20260619 00:56:13.561532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--webserver_interface=127.0.97.62
--webserver_port=43571
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:13.912912 15332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:13.913436 15332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:13.913650 15332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:13.921888 15332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:13.922030 15332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:13.922138 15332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:13.922251 15332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:13.922344 15332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:13.933665 15332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:13.935168 15332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:13.936626 15332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:13.949765 15338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:13.949806 15340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:13.950111 15337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:13.951072 15332 server_base.cc:1034] running on GCE node
I20260619 00:56:13.952031 15332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:13.953684 15332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:13.954973 15332 hybrid_clock.cc:648] HybridClock initialized: now 1781830573954902 us; error 45 us; skew 500 ppm
I20260619 00:56:13.955403 15332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:13 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:13.959020 15332 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:13.960929 15332 webserver.cc:460] Webserver started at http://127.0.97.62:43571/ using document root <none> and password file <none>
I20260619 00:56:13.961746 15332 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:13.961917 15332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:13.968400 15332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 00:56:13.972162 15347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:13.973937 15332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 00:56:13.974185 15332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "ab53d0096d0c408797a3fb950e03e5fb"
format_stamp: "Formatted at 2026-06-19 00:56:07 on dist-test-slave-r7bh"
server_key: "49f5923af0afb07f928e0a812199e4b8"
server_key_iv: "1f798e46a33143907c4301cecd8b0254"
server_key_version: "encryptionkey@0"
I20260619 00:56:13.975311 15332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:14.002909 15332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:14.008596 15332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:14.008966 15332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:14.029505 15332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37873
I20260619 00:56:14.029561 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37873 every 8 connection(s)
I20260619 00:56:14.031155 15332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:14.037328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15332
I20260619 00:56:14.038268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39271
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=42005
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:14.039294 15399 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:56:14.042907 15399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap starting.
I20260619 00:56:14.076234 15399 log.cc:824] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:14.099159 15399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:56:14.099932 15399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb: Bootstrap complete.
I20260619 00:56:14.115582 15399 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:14.116997 15399 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Initialized, Role: FOLLOWER
I20260619 00:56:14.117861 15399 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:14.118175 15399 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:14.118315 15399 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:14.118499 15399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 2 FOLLOWER]: Advancing to term 3
I20260619 00:56:14.124573 15399 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:14.125326 15399 leader_election.cc:302] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab53d0096d0c408797a3fb950e03e5fb; no voters:
I20260619 00:56:14.126036 15399 leader_election.cc:288] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 00:56:14.126227 15403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 3 FOLLOWER]: Leader election won for term 3
I20260619 00:56:14.127485 15403 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [term 3 LEADER]: Becoming Leader. State: Replica: ab53d0096d0c408797a3fb950e03e5fb, State: Running, Role: LEADER
I20260619 00:56:14.128105 15403 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } }
I20260619 00:56:14.128409 15399 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:14.130884 15405 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab53d0096d0c408797a3fb950e03e5fb. Latest consensus state: current_term: 3 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:14.130858 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ab53d0096d0c408797a3fb950e03e5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab53d0096d0c408797a3fb950e03e5fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37873 } } }
I20260619 00:56:14.131348 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:14.131352 15405 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab53d0096d0c408797a3fb950e03e5fb [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:14.146548 15409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:14.155776 15409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:14.158010 15409 catalog_manager.cc:1259] Loaded cluster ID: 592d6140578a4be9b73a659b607c7ee5
I20260619 00:56:14.158263 15409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 00:56:14.161612 15409 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f27268ec9ad google::LogMessage::Fail() at ??:0
@ 0x7f27268ef88a google::LogMessage::SendToLog() at ??:0
@ 0x7f27268ec4de google::LogMessage::Flush() at ??:0
@ 0x7f27268eda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f273c31f512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f273c31c976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f273c3a4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f273c3a4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2729538345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f272953b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f272953b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2729516eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f272c7f06db start_thread at ??:0
@ 0x7f272578661f clone at ??:0
W20260619 00:56:14.499262 15401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:14.499568 15401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:14.499668 15401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:14.507315 15401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:14.507440 15401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:14.507558 15401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:56:14.517781 15401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42005
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:14.519130 15401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:14.520466 15401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:14.534799 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:14.534961 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:14.534965 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:14.536073 15401 server_base.cc:1034] running on GCE node
I20260619 00:56:14.536761 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:14.537864 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:14.539175 15401 hybrid_clock.cc:648] HybridClock initialized: now 1781830574539092 us; error 42 us; skew 500 ppm
I20260619 00:56:14.539566 15401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:14 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:14.543146 15401 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:56:14.544770 15401 webserver.cc:460] Webserver started at http://127.0.97.1:42005/ using document root <none> and password file <none>
I20260619 00:56:14.545574 15401 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:14.545706 15401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:14.551775 15401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:56:14.554816 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:14.556529 15401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:56:14.556783 15401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "1cc5587acbd748d7ade71413bb09a48b"
format_stamp: "Formatted at 2026-06-19 00:56:08 on dist-test-slave-r7bh"
server_key: "25513c71ba9d259fedbe8c9b454b7cea"
server_key_iv: "25b5235e0db9b6db754ceb7b2d02be4c"
server_key_version: "encryptionkey@0"
I20260619 00:56:14.558071 15401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:14.580436 15401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:14.585323 15401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:14.585644 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:14.586762 15401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:14.588270 15401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:14.588419 15401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:14.588584 15401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:14.588668 15401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:14.626086 15401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39271
I20260619 00:56:14.626152 15562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39271 every 8 connection(s)
I20260619 00:56:14.627635 15401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:56:14.629719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15401
I20260619 00:56:14.630354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37393
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=46807
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:14.637642 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:14.630926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:51149)
0619 00:56:14.631838 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:14.632126 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":396,"threads_started":1}
W20260619 00:56:14.638558 15563 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37873: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
I20260619 00:56:14.639729 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:14.639236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:55683)
0619 00:56:14.639454 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:14.639558 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 00:56:14.640861 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:14.640379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:36631)
0619 00:56:14.640560 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:14.640664 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 00:56:14.641211 15563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 00:56:14.975627 15566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:14.975963 15566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:14.976065 15566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:14.983573 15566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:14.983732 15566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:14.983877 15566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:56:14.995260 15566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46807
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:14.996716 15566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:14.998231 15566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:15.014699 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:15.015007 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:15.014698 15572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:15.019098 15566 server_base.cc:1034] running on GCE node
I20260619 00:56:15.019969 15566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:15.021219 15566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:15.022502 15566 hybrid_clock.cc:648] HybridClock initialized: now 1781830575022446 us; error 55 us; skew 500 ppm
I20260619 00:56:15.022931 15566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:15 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:15.026751 15566 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:56:15.028586 15566 webserver.cc:460] Webserver started at http://127.0.97.2:46807/ using document root <none> and password file <none>
I20260619 00:56:15.029482 15566 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:15.029649 15566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:15.036044 15566 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:56:15.039294 15581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.040979 15566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:56:15.041224 15566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "17b621bd788a4194b1079ccd5f51ea69"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "0a6aea4d4f6b7c94451e62789b66f0ca"
server_key_iv: "57fa141120a59d36ec8f1ced35cd97dc"
server_key_version: "encryptionkey@0"
I20260619 00:56:15.042392 15566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:15.055217 15566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:15.060221 15566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:15.060482 15566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:15.061617 15566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:15.063208 15566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:15.063295 15566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.063452 15566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:15.063555 15566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.098464 15566 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37393
I20260619 00:56:15.098565 15693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37393 every 8 connection(s)
I20260619 00:56:15.100131 15566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:56:15.105443 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.103416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:60535)
0619 00:56:15.104219 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.104429 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":314,"threads_started":1}
W20260619 00:56:15.106156 15694 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37873: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
I20260619 00:56:15.106782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15566
I20260619 00:56:15.107126 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.106759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:43525)
0619 00:56:15.106934 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.107015 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 00:56:15.107553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35153
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37873
--webserver_port=41395
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:15.108204 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.107749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:50631)
0619 00:56:15.107940 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.108042 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 00:56:15.108520 15694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 00:56:15.447625 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:15.447988 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:15.448153 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:15.455632 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:15.455766 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:15.455960 15697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:56:15.466200 15697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41395
--tserver_master_addrs=127.0.97.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:15.467674 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:15.469096 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:15.483574 15702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:15.483651 15705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:15.483551 15703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:15.484792 15697 server_base.cc:1034] running on GCE node
I20260619 00:56:15.485545 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:15.486588 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:15.487867 15697 hybrid_clock.cc:648] HybridClock initialized: now 1781830575487805 us; error 58 us; skew 500 ppm
I20260619 00:56:15.488253 15697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:15 dist-test-slave-r7bh krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:15.491734 15697 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:56:15.493516 15697 webserver.cc:460] Webserver started at http://127.0.97.3:41395/ using document root <none> and password file <none>
I20260619 00:56:15.494331 15697 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:15.494503 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:15.500720 15697 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:56:15.503816 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.505453 15697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:56:15.505667 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "959b15f0d7f54d7ba2a3bd8d6225aac8"
format_stamp: "Formatted at 2026-06-19 00:56:09 on dist-test-slave-r7bh"
server_key: "8b2f492fd4729e8734542ce2fd99fb11"
server_key_iv: "a89585489745bf379a0e33c47dcfc5f2"
server_key_version: "encryptionkey@0"
I20260619 00:56:15.506768 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:15.530678 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:15.535845 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:15.536157 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:15.537336 15697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:15.538767 15697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:15.538892 15697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.539026 15697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:15.539111 15697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:15.574965 15697 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35153
I20260619 00:56:15.575067 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35153 every 8 connection(s)
I20260619 00:56:15.576578 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:56:15.581876 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.579820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:37681)
0619 00:56:15.580571 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.580786 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260619 00:56:15.582270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15697
W20260619 00:56:15.582723 15825 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37873 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37873: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
I20260619 00:56:15.583634 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.583290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:49699)
0619 00:56:15.583448 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.583522 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 00:56:15.584558 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.584225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:40049)
0619 00:56:15.584367 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.584442 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 00:56:15.584851 15825 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 00:56:15.642586 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:15.641811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:53443)
0619 00:56:15.642255 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:15.642345 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260619 00:56:16.109903 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:16.109081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:35655)
0619 00:56:16.109546 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:16.109652 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":216,"threads_started":1}
I20260619 00:56:16.586310 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:16.585464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:55709)
0619 00:56:16.585970 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:16.586065 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
I20260619 00:56:16.644495 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:16.643673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:42215)
0619 00:56:16.644221 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:16.644321 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":283,"threads_started":1}
I20260619 00:56:17.111963 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:17.111104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:44017)
0619 00:56:17.111685 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:17.111774 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
I20260619 00:56:17.588173 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:17.587412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:32883)
0619 00:56:17.587911 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:17.588001 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":232,"threads_started":1}
I20260619 00:56:17.646375 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:17.645589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:54911)
0619 00:56:17.646101 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:17.646192 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
I20260619 00:56:18.113798 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:18.112965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:32901)
0619 00:56:18.113452 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:18.113587 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260619 00:56:18.589936 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:18.589171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:38213)
0619 00:56:18.589662 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:18.589760 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260619 00:56:18.648279 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:18.647393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:43065)
0619 00:56:18.647949 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:18.648059 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
I20260619 00:56:19.115556 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:19.114801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:55073)
0619 00:56:19.115234 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:19.115354 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":209,"threads_started":1}
I20260619 00:56:19.591888 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:19.591017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:47301)
0619 00:56:19.591585 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:19.591678 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":302,"threads_started":1}
I20260619 00:56:19.650053 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:19.649325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:44867)
0619 00:56:19.649793 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:19.649877 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260619 00:56:20.117493 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:20.116627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:45397)
0619 00:56:20.117194 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:20.117311 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":333,"threads_started":1}
I20260619 00:56:20.594136 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:20.593285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:39341)
0619 00:56:20.593839 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:20.593933 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":281,"threads_started":1}
I20260619 00:56:20.652474 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:20.651382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:36497)
0619 00:56:20.651961 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:20.652131 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":267,"threads_started":1}
I20260619 00:56:21.119199 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:21.118453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:38803)
0619 00:56:21.118923 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:21.119017 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260619 00:56:21.596222 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:21.595345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:56645)
0619 00:56:21.595899 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:21.596023 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":300,"threads_started":1}
I20260619 00:56:21.654165 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:21.653399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:44581)
0619 00:56:21.653880 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:21.653958 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":211,"threads_started":1}
I20260619 00:56:22.121260 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:22.120300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:45555)
0619 00:56:22.120927 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:22.121027 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":247,"threads_started":1}
I20260619 00:56:22.598220 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:22.597337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:51493)
0619 00:56:22.597896 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:22.597996 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":233,"threads_started":1}
I20260619 00:56:22.656224 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:22.655325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:54893)
0619 00:56:22.655801 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:22.655895 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260619 00:56:23.122946 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:23.122185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:34239)
0619 00:56:23.122680 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:23.122773 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260619 00:56:23.600157 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:23.599380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:59517)
0619 00:56:23.599889 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:23.599987 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260619 00:56:23.658303 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:23.657405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:42225)
0619 00:56:23.657963 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:23.658088 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":302,"threads_started":1}
I20260619 00:56:24.124666 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:24.123919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:59177)
0619 00:56:24.124396 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:24.124487 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":234,"threads_started":1}
I20260619 00:56:24.602051 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:24.601275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:57079)
0619 00:56:24.601764 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:24.601859 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":229,"threads_started":1}
I20260619 00:56:24.660072 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:24.659370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:37585)
0619 00:56:24.659819 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:24.659907 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260619 00:56:25.126408 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:25.125687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:36057)
0619 00:56:25.126153 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:25.126238 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260619 00:56:25.603832 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:25.603057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:58207)
0619 00:56:25.603572 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:25.603662 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":276,"threads_started":1}
I20260619 00:56:25.661840 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:25.661067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:56011)
0619 00:56:25.661509 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:25.661604 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":204,"threads_started":1}
I20260619 00:56:26.128217 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:26.127391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:42619)
0619 00:56:26.127957 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:26.128038 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":325,"threads_started":1}
I20260619 00:56:26.605860 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:26.605072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:33337)
0619 00:56:26.605591 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:26.605682 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":305,"threads_started":1}
I20260619 00:56:26.663507 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:26.662789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:49361)
0619 00:56:26.663237 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:26.663350 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260619 00:56:27.130079 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:27.129367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:36893)
0619 00:56:27.129828 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:27.129915 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260619 00:56:27.607486 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:27.606782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:49531)
0619 00:56:27.607231 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:27.607324 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260619 00:56:27.665269 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:27.664439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:43245)
0619 00:56:27.664977 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:27.665069 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":227,"threads_started":1}
I20260619 00:56:28.131771 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:28.131019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:51227)
0619 00:56:28.131501 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:28.131589 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":227,"threads_started":1}
I20260619 00:56:28.609201 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:28.608421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:48745)
0619 00:56:28.608917 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:28.609009 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260619 00:56:28.666863 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:28.666128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:56631)
0619 00:56:28.666573 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:28.666657 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":255,"threads_started":1}
I20260619 00:56:29.133460 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:29.132634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:43061)
0619 00:56:29.133193 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:29.133277 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":274,"threads_started":1}
I20260619 00:56:29.610930 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:29.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:46713)
0619 00:56:29.610619 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:29.610707 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260619 00:56:29.668733 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:29.668024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:41299)
0619 00:56:29.668463 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:29.668548 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260619 00:56:30.135313 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.134607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.2:51639)
0619 00:56:30.135046 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:30.135135 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260619 00:56:30.593706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15401
I20260619 00:56:30.613102 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.612223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.3:45701)
0619 00:56:30.612694 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:30.612837 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260619 00:56:30.617573 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.595025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.0.97.1:39271)
0619 00:56:30.595520 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:30.595539 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:30.595586 (+ 47us) server_negotiation.cc:407] Connection header received
0619 00:56:30.597112 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:30.597123 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:30.597565 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:30.597860 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:30.599253 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.600550 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.601884 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.602568 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.602710 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:30.610750 (+ 8040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:30.610759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:30.610777 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:30.610822 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:30.614478 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.615201 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.615208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.615215 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.615397 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.615736 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.615739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.615741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.616271 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:30.616520 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:30.616926 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 00:56:30.617281 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":332,"threads_started":1}
I20260619 00:56:30.619089 15558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:30.711660 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.710442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37873 (local address 127.0.97.1:46167)
0619 00:56:30.711141 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:30.711324 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":394,"threads_started":1}
I20260619 00:56:30.711869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15401
I20260619 00:56:30.736992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15566
I20260619 00:56:30.755434 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.738104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.0.97.2:37393)
0619 00:56:30.738659 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:30.738677 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:30.738723 (+ 46us) server_negotiation.cc:407] Connection header received
0619 00:56:30.740909 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:30.740922 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:30.741366 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:30.741666 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:30.742953 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.744259 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.745755 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.746614 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.746757 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:30.749366 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:30.749374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:30.749390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:30.749428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:30.752130 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.752792 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.752840 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.752853 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.753036 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.753403 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.753408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.753411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.753923 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:30.754224 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:30.754709 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 00:56:30.755133 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260619 00:56:30.757197 15689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:30.843834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15566
I20260619 00:56:30.861229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15697
I20260619 00:56:30.879416 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:30.862272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.97.3:35153)
0619 00:56:30.862826 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:30.862841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:30.862882 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:56:30.864496 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:30.864505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:30.865015 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:30.865390 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:30.866843 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.868077 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.869478 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:30.870153 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:30.870384 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:30.872998 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:30.873006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:30.873021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:30.873062 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:30.875804 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.876673 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.876680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.876686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.876929 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:30.877309 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:30.877314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:30.877317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:30.877869 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:30.878173 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:30.878683 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 00:56:30.879114 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":257,"threads_started":1}
I20260619 00:56:30.880893 15820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:30.963043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15697
2026-06-19T00:56:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24778 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:56:31 dist-test-slave-r7bh krb5kdc[15883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:56:31 dist-test-slave-r7bh krb5kdc[15883](info): set up 2 sockets
Jun 19 00:56:31 dist-test-slave-r7bh krb5kdc[15883](info): commencing operation
krb5kdc: starting...
W20260619 00:56:32.065524 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:56:32 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:56:32.311980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:32.647356 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:32.647666 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:32.647837 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:32.655316 15896 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 00:56:32.655426 15896 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 00:56:32.655465 15896 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 00:56:32.655567 15896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:32.655661 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:32.655782 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:32.655858 15896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:32.655937 15896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:32.666332 15896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:32.667699 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:32.669859 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:32.682369 15901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:32.682292 15902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:32.682473 15904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:32.683166 15896 server_base.cc:1034] running on GCE node
I20260619 00:56:32.684099 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:32.685736 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:32.687032 15896 hybrid_clock.cc:648] HybridClock initialized: now 1781830592686970 us; error 43 us; skew 500 ppm
I20260619 00:56:32.687407 15896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:32 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:32.690683 15896 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:32.692222 15896 webserver.cc:460] Webserver started at http://127.0.97.62:41047/ using document root <none> and password file <none>
I20260619 00:56:32.693029 15896 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:32.693161 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:32.693490 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:32.696738 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "f38e91ccc0974d349658bd98f5b4ec3a"
format_stamp: "Formatted at 2026-06-19 00:56:32 on dist-test-slave-r7bh"
server_key: "461f353b5948c424365048c9ef52becc"
server_key_iv: "07aed39d9cd29a8f8717b3f350e36402"
server_key_version: "encryptionkey@0"
I20260619 00:56:32.697599 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "f38e91ccc0974d349658bd98f5b4ec3a"
format_stamp: "Formatted at 2026-06-19 00:56:32 on dist-test-slave-r7bh"
server_key: "461f353b5948c424365048c9ef52becc"
server_key_iv: "07aed39d9cd29a8f8717b3f350e36402"
server_key_version: "encryptionkey@0"
I20260619 00:56:32.703775 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260619 00:56:32.707930 15911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:32.709422 15896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:56:32.709623 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "f38e91ccc0974d349658bd98f5b4ec3a"
format_stamp: "Formatted at 2026-06-19 00:56:32 on dist-test-slave-r7bh"
server_key: "461f353b5948c424365048c9ef52becc"
server_key_iv: "07aed39d9cd29a8f8717b3f350e36402"
server_key_version: "encryptionkey@0"
I20260619 00:56:32.709856 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:32.725135 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:32.729877 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:32.730195 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:32.749305 15896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37241
I20260619 00:56:32.749311 15962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37241 every 8 connection(s)
I20260619 00:56:32.750765 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:32.754549 15963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:32.755647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15896
I20260619 00:56:32.755863 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:56:32.756501 388 external_mini_cluster.cc:1428] Setting key 6c351f117362ee0e1c7a62e3c57894e6
I20260619 00:56:32.768996 15963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap starting.
Jun 19 00:56:32 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:32.773509 15963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:32.774981 15963 log.cc:824] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:32.778251 15963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: No bootstrap required, opened a new log
I20260619 00:56:32.782445 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:32.758597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.0.97.62:37241)
0619 00:56:32.759524 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:32.759539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:32.759602 (+ 63us) server_negotiation.cc:407] Connection header received
0619 00:56:32.761389 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:32.761411 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:32.761908 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:32.762323 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:32.763784 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:32.765199 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:32.767365 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:32.768047 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:32.768255 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:32.774007 (+ 5752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:32.774022 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:32.774041 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:32.774126 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:32.778084 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:32.778900 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:32.778909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:32.778920 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:32.779094 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:32.779501 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:32.779507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:32.779511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:32.780088 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:32.780434 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:32.780949 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 00:56:32.781391 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":357,"threads_started":1}
I20260619 00:56:32.786615 15963 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:32.787142 15963 raft_consensus.cc:374] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:32.787262 15963 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Initialized, Role: FOLLOWER
I20260619 00:56:32.787952 15963 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:32.788234 15963 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:32.788372 15963 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:32.788544 15963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:32.790870 15963 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:32.791412 15963 leader_election.cc:302] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38e91ccc0974d349658bd98f5b4ec3a; no voters:
I20260619 00:56:32.792007 15963 leader_election.cc:288] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:56:32.792143 15968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:32.792886 15968 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 LEADER]: Becoming Leader. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Running, Role: LEADER
I20260619 00:56:32.793401 15968 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:32.794188 15963 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:32.796125 15970 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38e91ccc0974d349658bd98f5b4ec3a. Latest consensus state: current_term: 1 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:56:32.796396 15969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:56:32.796588 15970 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:32.796645 15969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:32.797508 15973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:32.803905 15973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:32.813706 15973 catalog_manager.cc:1347] Generated new cluster ID: 1b64615b24e24d2ab54ccd8fc9e04c3a
I20260619 00:56:32.813844 15973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:32.838759 15973 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:56:32.840741 15973 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:32.850332 15973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Generated new TSK 0
I20260619 00:56:32.851420 15973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:56:32.920635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:33.261214 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:33.261565 15989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:33.261667 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:33.268920 15989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:33.269037 15989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:33.269152 15989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:56:33.278955 15989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:33.280174 15989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:33.281458 15989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:33.294724 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:33.294996 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:33.294792 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:33.296279 15989 server_base.cc:1034] running on GCE node
I20260619 00:56:33.297111 15989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:33.298125 15989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:33.299360 15989 hybrid_clock.cc:648] HybridClock initialized: now 1781830593299305 us; error 57 us; skew 500 ppm
I20260619 00:56:33.299727 15989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:33 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:33.303050 15989 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:56:33.304605 15989 webserver.cc:460] Webserver started at http://127.0.97.1:36545/ using document root <none> and password file <none>
I20260619 00:56:33.305358 15989 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:33.305502 15989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:33.305828 15989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:33.309038 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "db56829b3b68aaa3108d308f2aaca038"
server_key_iv: "a3b9068bbcb9c008d9da4250ed3fda63"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.309834 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "db56829b3b68aaa3108d308f2aaca038"
server_key_iv: "a3b9068bbcb9c008d9da4250ed3fda63"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.315424 15989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:56:33.318933 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.320307 15989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:56:33.320469 15989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "db56829b3b68aaa3108d308f2aaca038"
server_key_iv: "a3b9068bbcb9c008d9da4250ed3fda63"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.320665 15989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:33.342944 15989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:33.347343 15989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:33.347642 15989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:33.348651 15989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:33.350219 15989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:33.350339 15989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.350488 15989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:33.350569 15989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.383379 15989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35471
I20260619 00:56:33.383446 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35471 every 8 connection(s)
I20260619 00:56:33.384924 15989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:56:33.394052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 15989
I20260619 00:56:33.394276 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:56:33.394992 388 external_mini_cluster.cc:1428] Setting key f17ca8b1114280893aa71aa500868a12
Jun 19 00:56:33 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:33.405143 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:33.387621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43173 (local address 127.0.97.62:37241)
0619 00:56:33.388131 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:33.388138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:33.389081 (+ 943us) server_negotiation.cc:407] Connection header received
0619 00:56:33.390864 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:33.390868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:33.390938 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:33.391036 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:33.392495 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.393272 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.395033 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.395682 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.395858 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:33.399342 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:33.399348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:33.399351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:33.399385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:33.402092 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:33.402851 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:33.402863 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:33.402867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:33.402985 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:33.403428 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:33.403434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:33.403437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:33.403700 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:33.403862 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:33.404639 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 00:56:33.404874 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260619 00:56:33.406709 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:33.388101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:43173)
0619 00:56:33.388901 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:33.388967 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:33.390601 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:33.391242 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:33.391251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:33.391761 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:33.392333 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:33.392350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.393427 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.393433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:33.394877 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:33.394888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.395976 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.395985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:33.396233 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:33.396949 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:33.396972 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:33.399113 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:33.402245 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:33.402255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:33.402274 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:33.402718 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:33.403105 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:33.403110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:33.403114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:33.403306 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:33.403829 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:33.403837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:33.404210 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 00:56:33.404540 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 00:56:33.405169 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":293,"threads_started":1}
I20260619 00:56:33.408341 16117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:33.408674 16117 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:33.409466 16117 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:33.411645 15928 ts_manager.cc:194] Registered new tserver with Master: bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471)
I20260619 00:56:33.413288 15928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43173
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:56:33.448323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:33.773733 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:33.774039 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:33.774140 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:33.781649 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:33.781770 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:33.781891 16123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:56:33.791781 16123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:33.793063 16123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:33.794379 16123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:33.807482 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:33.807497 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:33.807721 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:33.808370 16123 server_base.cc:1034] running on GCE node
I20260619 00:56:33.809010 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:33.809917 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:33.811121 16123 hybrid_clock.cc:648] HybridClock initialized: now 1781830593811075 us; error 34 us; skew 500 ppm
I20260619 00:56:33.811455 16123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:33 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:33.814640 16123 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:56:33.816144 16123 webserver.cc:460] Webserver started at http://127.0.97.2:45319/ using document root <none> and password file <none>
I20260619 00:56:33.816882 16123 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:33.817024 16123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:33.817338 16123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:33.820349 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "01edb8fe93604d3788a60834c2a26e4d"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "edcf784baf47652fa0147e27e37df43d"
server_key_iv: "a0684dd9396bf2dae7a8c02b843211b0"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.821195 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "01edb8fe93604d3788a60834c2a26e4d"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "edcf784baf47652fa0147e27e37df43d"
server_key_iv: "a0684dd9396bf2dae7a8c02b843211b0"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.826491 16123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:56:33.829970 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.831362 16123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:56:33.831543 16123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "01edb8fe93604d3788a60834c2a26e4d"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "edcf784baf47652fa0147e27e37df43d"
server_key_iv: "a0684dd9396bf2dae7a8c02b843211b0"
server_key_version: "encryptionkey@0"
I20260619 00:56:33.831749 16123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:33.865587 16123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:33.869791 16123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:33.870095 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:33.871090 16123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:33.872393 16123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:33.872496 16123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.872649 16123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:33.872721 16123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:33.906829 16123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39387
I20260619 00:56:33.906885 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39387 every 8 connection(s)
I20260619 00:56:33.908325 16123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:56:33.911837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16123
I20260619 00:56:33.912039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:56:33.912550 388 external_mini_cluster.cc:1428] Setting key c7e55261856d4f058a3e540dc957de17
Jun 19 00:56:33 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:33.930154 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:33.910960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41683 (local address 127.0.97.62:37241)
0619 00:56:33.911428 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:33.911433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:33.912757 (+ 1324us) server_negotiation.cc:407] Connection header received
0619 00:56:33.914706 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:33.914711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:33.914793 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:33.914928 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:33.916441 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.917644 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.919212 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.919995 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.920134 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:33.924141 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:33.924147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:33.924150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:33.924182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:33.926918 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:33.927824 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:33.927828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:33.927830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:33.927965 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:33.928433 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:33.928436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:33.928438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:33.928684 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:33.928790 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:33.929709 (+ 919us) server_negotiation.cc:299] Negotiation successful
0619 00:56:33.929886 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260619 00:56:33.931727 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:33.911579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:41683)
0619 00:56:33.912413 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:33.912458 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:33.914388 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:33.915140 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:33.915149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:33.915691 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:33.916251 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:33.916269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.917801 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.917805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:33.919058 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:33.919070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:33.920209 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:33.920214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:33.920422 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:33.921191 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:33.921218 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:33.923902 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:33.927058 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:33.927070 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:33.927090 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:33.927713 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:33.928068 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:33.928076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:33.928083 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:33.928329 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:33.928835 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:33.928846 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:33.929327 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 00:56:33.929659 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 00:56:33.930231 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":352,"threads_started":1}
I20260619 00:56:33.933313 16251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:33.933643 16251 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:33.934393 16251 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:33.935912 15928 ts_manager.cc:194] Registered new tserver with Master: 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387)
I20260619 00:56:33.936865 15928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41683
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:56:33.967578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:34.308683 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:34.309098 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:34.309211 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:34.316851 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:34.316989 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:34.317107 16257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:56:34.327325 16257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:34.328636 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:34.330060 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:34.343269 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:34.343353 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:34.343381 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:34.344400 16257 server_base.cc:1034] running on GCE node
I20260619 00:56:34.345106 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:34.346153 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:34.347363 16257 hybrid_clock.cc:648] HybridClock initialized: now 1781830594347292 us; error 74 us; skew 500 ppm
I20260619 00:56:34.347736 16257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:34 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:34.351032 16257 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:56:34.352761 16257 webserver.cc:460] Webserver started at http://127.0.97.3:34171/ using document root <none> and password file <none>
I20260619 00:56:34.353519 16257 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:34.353672 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:34.354004 16257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:56:34.357100 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "542b629aaff64ace8620660481132eda"
format_stamp: "Formatted at 2026-06-19 00:56:34 on dist-test-slave-r7bh"
server_key: "3acc2316b7d23ef14dc3c030ea609bba"
server_key_iv: "ceb2025bd655ef9a0405f47cb1ea379e"
server_key_version: "encryptionkey@0"
I20260619 00:56:34.357908 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "542b629aaff64ace8620660481132eda"
format_stamp: "Formatted at 2026-06-19 00:56:34 on dist-test-slave-r7bh"
server_key: "3acc2316b7d23ef14dc3c030ea609bba"
server_key_iv: "ceb2025bd655ef9a0405f47cb1ea379e"
server_key_version: "encryptionkey@0"
I20260619 00:56:34.363363 16257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 00:56:34.367272 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:34.368916 16257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:56:34.369130 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "542b629aaff64ace8620660481132eda"
format_stamp: "Formatted at 2026-06-19 00:56:34 on dist-test-slave-r7bh"
server_key: "3acc2316b7d23ef14dc3c030ea609bba"
server_key_iv: "ceb2025bd655ef9a0405f47cb1ea379e"
server_key_version: "encryptionkey@0"
I20260619 00:56:34.369382 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:34.388376 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:34.392597 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:34.392966 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:34.393946 16257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:34.395463 16257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:34.395572 16257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:34.395730 16257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:34.395808 16257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:34.416679 16117 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:34.431813 16257 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36777
I20260619 00:56:34.431912 16384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36777 every 8 connection(s)
I20260619 00:56:34.433226 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:56:34.441591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16257
I20260619 00:56:34.441804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:56:34.442269 388 external_mini_cluster.cc:1428] Setting key 10e6093c9df814db67e9ea1ac04ab190
Jun 19 00:56:34 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:34.453626 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.435901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54867 (local address 127.0.97.62:37241)
0619 00:56:34.436405 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.436410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.437536 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 00:56:34.439504 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.439510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.439616 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.439737 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:34.441359 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.442227 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.444062 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.444493 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.444640 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.448199 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:34.448204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:34.448207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:34.448241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:34.450735 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.451432 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.451436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.451438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.451555 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.451971 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.451975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.451977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.452297 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:34.452421 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.453148 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.453328 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260619 00:56:34.454596 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.436395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:54867)
0619 00:56:34.437341 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.437389 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.439207 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.439990 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.439999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.440534 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:34.441164 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.441182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.442395 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.442399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.443901 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.443913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.444730 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.444738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.445047 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.445768 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:34.445796 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:34.448000 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:34.450860 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:34.450867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:34.450880 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:34.451311 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:34.451645 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:34.451648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:34.451651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:34.451826 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:34.452405 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:34.452411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:34.452728 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.453037 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.453454 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":470,"threads_started":1}
I20260619 00:56:34.456162 16385 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:34.456524 16385 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:34.457381 16385 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:34.458777 15928 ts_manager.cc:194] Registered new tserver with Master: 542b629aaff64ace8620660481132eda (127.0.97.3:36777)
I20260619 00:56:34.459558 15928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54867
I20260619 00:56:34.470572 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:56:34.488991 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.475584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.0.97.62:37241)
0619 00:56:34.475758 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.475762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.476193 (+ 431us) server_negotiation.cc:407] Connection header received
0619 00:56:34.476340 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.476344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.476431 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.476522 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:34.478163 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.478944 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.480339 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.480987 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.481106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.483535 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:34.483540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:34.483544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:34.483579 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:34.486400 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.487186 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.487190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.487193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.487344 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.487700 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.487703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.487705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.488032 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:34.488228 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.488492 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.488680 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:56:34.493628 15928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:56:34.501627 15928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:56:34.548175 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.515669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36777 (local address 127.0.0.1:46478)
0619 00:56:34.533241 (+ 17572us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.533324 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.533629 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.536914 (+ 3285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.536926 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.536968 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:34.537461 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.537476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.539974 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.539980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.543567 (+ 3587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.543593 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.547000 (+ 3407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.547007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.547231 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.547297 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.547442 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.547654 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17123,"mutex_wait_us":136,"thread_start_us":19128,"threads_started":1}
I20260619 00:56:34.549480 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.516028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.97.3:36777)
0619 00:56:34.535645 (+ 19617us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.535655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.535688 (+ 33us) server_negotiation.cc:407] Connection header received
0619 00:56:34.535789 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.535798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.536244 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.536500 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:34.537722 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.539759 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.543862 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.546231 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.546387 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.546757 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.548684 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.548988 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19333,"thread_start_us":19176,"threads_started":1}
I20260619 00:56:34.549467 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.516631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.97.2:39387)
0619 00:56:34.531410 (+ 14779us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.531424 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.535255 (+ 3831us) server_negotiation.cc:407] Connection header received
0619 00:56:34.535673 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.535688 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.536114 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.536435 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:34.538085 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.540369 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.544622 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.547039 (+ 2417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.547213 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.547521 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.548569 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.548927 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14511,"thread_start_us":14355,"threads_started":1}
I20260619 00:56:34.550755 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.515985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39387 (local address 127.0.0.1:43396)
0619 00:56:34.535098 (+ 19113us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.535131 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.535409 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.537010 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.537020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.537111 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:34.537773 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.537793 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.540697 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.540705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.544301 (+ 3596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.544328 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.547857 (+ 3529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.547871 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.548243 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.548302 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.549880 (+ 1578us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.550143 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18942,"mutex_wait_us":4,"spinlock_wait_cycles":8064,"thread_start_us":17979,"threads_started":1}
I20260619 00:56:34.554944 16320 tablet_service.cc:1467] Processing CreateTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf (DEFAULT_TABLE table=test-table [id=324d48ae6e6c4e04908e2738e273eb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:34.555330 16186 tablet_service.cc:1467] Processing CreateTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf (DEFAULT_TABLE table=test-table [id=324d48ae6e6c4e04908e2738e273eb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:34.556905 16320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deabca67894d8a9e73a8d1ba7712bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:34.556912 16186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deabca67894d8a9e73a8d1ba7712bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:34.558926 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.535737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:41736)
0619 00:56:34.551224 (+ 15487us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.551250 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.551402 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.552342 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.552347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.552381 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:34.552599 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.552612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.554317 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.554326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.555624 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.555640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.557965 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.557971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.558201 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.558263 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.558386 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.558565 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15369,"thread_start_us":15676,"threads_started":1}
I20260619 00:56:34.558934 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.535567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.0.97.1:35471)
0619 00:56:34.550915 (+ 15348us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.550925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.551580 (+ 655us) server_negotiation.cc:407] Connection header received
0619 00:56:34.551745 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.551754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.552071 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.552776 (+ 705us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:34.553076 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.554148 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.555939 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.557782 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.557952 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.558208 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.558441 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.558607 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15090,"thread_start_us":15403,"threads_started":1}
I20260619 00:56:34.562973 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf (DEFAULT_TABLE table=test-table [id=324d48ae6e6c4e04908e2738e273eb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:34.564553 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97deabca67894d8a9e73a8d1ba7712bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:34.573783 16404 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Bootstrap starting.
I20260619 00:56:34.577677 16405 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Bootstrap starting.
I20260619 00:56:34.578182 16404 tablet_bootstrap.cc:654] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:34.579459 16406 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Bootstrap starting.
I20260619 00:56:34.579962 16404 log.cc:824] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:34.582041 16405 tablet_bootstrap.cc:654] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:34.583343 16406 tablet_bootstrap.cc:654] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:34.583458 16405 log.cc:824] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:34.584046 16404 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: No bootstrap required, opened a new log
I20260619 00:56:34.584475 16404 ts_tablet_manager.cc:1397] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 00:56:34.584677 16406 log.cc:824] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:34.586865 16405 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: No bootstrap required, opened a new log
I20260619 00:56:34.587164 16405 ts_tablet_manager.cc:1397] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 00:56:34.587931 16406 tablet_bootstrap.cc:492] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: No bootstrap required, opened a new log
I20260619 00:56:34.588209 16406 ts_tablet_manager.cc:1397] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 00:56:34.592669 16404 raft_consensus.cc:348] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.593194 16404 raft_consensus.cc:374] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:34.593353 16404 raft_consensus.cc:729] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542b629aaff64ace8620660481132eda, State: Initialized, Role: FOLLOWER
I20260619 00:56:34.594105 16404 consensus_queue.cc:260] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.595408 16385 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:34.596101 16404 ts_tablet_manager.cc:1428] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260619 00:56:34.598629 16405 raft_consensus.cc:348] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.599153 16405 raft_consensus.cc:374] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:34.599282 16405 raft_consensus.cc:729] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01edb8fe93604d3788a60834c2a26e4d, State: Initialized, Role: FOLLOWER
I20260619 00:56:34.599372 16406 raft_consensus.cc:348] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.599948 16406 raft_consensus.cc:374] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:34.600088 16406 raft_consensus.cc:729] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe6bc1fd98f4ba1b0a906eae9edfbcc, State: Initialized, Role: FOLLOWER
I20260619 00:56:34.599994 16405 consensus_queue.cc:260] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.600927 16406 consensus_queue.cc:260] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.603426 16406 ts_tablet_manager.cc:1428] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 00:56:34.604667 16251 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:34.605296 16405 ts_tablet_manager.cc:1428] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
W20260619 00:56:34.637450 16118 tablet.cc:2369] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:34.660540 16252 tablet.cc:2369] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:56:34.684976 16386 tablet.cc:2369] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:56:34.747032 16411 raft_consensus.cc:482] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:56:34.747424 16411 raft_consensus.cc:504] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.749315 16411 leader_election.cc:288] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 542b629aaff64ace8620660481132eda (127.0.97.3:36777), bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471)
I20260619 00:56:34.758496 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.751006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51001 (local address 127.0.97.1:35471)
0619 00:56:34.751427 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.751431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.751447 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:56:34.751510 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.751514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.751573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.751682 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:34.752419 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.753387 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.756056 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.757384 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.757529 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.757636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.758087 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.758242 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260619 00:56:34.758596 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.749602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.97.2:51001)
0619 00:56:34.750897 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.750924 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.751117 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.751828 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.751832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.751859 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:34.752252 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.752264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.753518 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.753525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.755833 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.755850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.757616 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.757622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.757891 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.757932 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.758038 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.758190 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":27,"thread_start_us":1577,"threads_started":1}
I20260619 00:56:34.758613 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.749965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42525 (local address 127.0.97.3:36777)
0619 00:56:34.750220 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.750229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.751659 (+ 1430us) server_negotiation.cc:407] Connection header received
0619 00:56:34.751819 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.751825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.751905 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.752013 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:34.752655 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.753775 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.756144 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.757409 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.757529 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.757636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.758217 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.758365 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:56:34.759255 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.749711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36777 (local address 127.0.97.2:42525)
0619 00:56:34.751524 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:34.751542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:34.751683 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:34.752035 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:34.752040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:34.752066 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:34.752480 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.752494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.753911 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.753916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.755976 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:34.755991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.757801 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.757806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:34.758025 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.758063 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:56:34.758791 (+ 728us) client_negotiation.cc:240] Negotiation successful
0619 00:56:34.758926 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"mutex_wait_us":36,"thread_start_us":730,"threads_started":1}
I20260619 00:56:34.760025 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deabca67894d8a9e73a8d1ba7712bf" candidate_uuid: "01edb8fe93604d3788a60834c2a26e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" is_pre_election: true
I20260619 00:56:34.760272 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deabca67894d8a9e73a8d1ba7712bf" candidate_uuid: "01edb8fe93604d3788a60834c2a26e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542b629aaff64ace8620660481132eda" is_pre_election: true
I20260619 00:56:34.760589 16072 raft_consensus.cc:2454] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01edb8fe93604d3788a60834c2a26e4d in term 0.
I20260619 00:56:34.760787 16340 raft_consensus.cc:2454] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01edb8fe93604d3788a60834c2a26e4d in term 0.
I20260619 00:56:34.761417 16140 leader_election.cc:302] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01edb8fe93604d3788a60834c2a26e4d, 542b629aaff64ace8620660481132eda, bfe6bc1fd98f4ba1b0a906eae9edfbcc; no voters:
I20260619 00:56:34.761898 16411 raft_consensus.cc:2781] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:56:34.762055 16411 raft_consensus.cc:482] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:56:34.762156 16411 raft_consensus.cc:3037] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:34.765450 16411 raft_consensus.cc:504] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.766289 16411 leader_election.cc:288] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [CANDIDATE]: Term 1 election: Requested vote from peers 542b629aaff64ace8620660481132eda (127.0.97.3:36777), bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471)
I20260619 00:56:34.767460 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deabca67894d8a9e73a8d1ba7712bf" candidate_uuid: "01edb8fe93604d3788a60834c2a26e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
I20260619 00:56:34.767728 16072 raft_consensus.cc:3037] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:34.767184 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97deabca67894d8a9e73a8d1ba7712bf" candidate_uuid: "01edb8fe93604d3788a60834c2a26e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542b629aaff64ace8620660481132eda"
I20260619 00:56:34.770467 16340 raft_consensus.cc:3037] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:34.771579 16072 raft_consensus.cc:2454] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01edb8fe93604d3788a60834c2a26e4d in term 1.
I20260619 00:56:34.772258 16140 leader_election.cc:302] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01edb8fe93604d3788a60834c2a26e4d, bfe6bc1fd98f4ba1b0a906eae9edfbcc; no voters:
I20260619 00:56:34.772626 16411 raft_consensus.cc:2781] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:34.773319 16411 raft_consensus.cc:686] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 1 LEADER]: Becoming Leader. State: Replica: 01edb8fe93604d3788a60834c2a26e4d, State: Running, Role: LEADER
I20260619 00:56:34.774111 16340 raft_consensus.cc:2454] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01edb8fe93604d3788a60834c2a26e4d in term 1.
I20260619 00:56:34.774065 16411 consensus_queue.cc:237] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } }
I20260619 00:56:34.779309 15927 catalog_manager.cc:5495] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "01edb8fe93604d3788a60834c2a26e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260619 00:56:34.827193 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.821096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.0.97.2:39387)
0619 00:56:34.821325 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.821330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.821349 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:56:34.821449 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.821453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.821533 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.821660 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:34.822053 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.822933 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.824682 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.825264 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.825372 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.825873 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:34.826410 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:56:34.826532 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.826746 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.826887 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:56:34.835983 16340 raft_consensus.cc:1261] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Refusing update from remote peer 01edb8fe93604d3788a60834c2a26e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:34.835945 16072 raft_consensus.cc:1261] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Refusing update from remote peer 01edb8fe93604d3788a60834c2a26e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:34.837044 16415 consensus_queue.cc:1035] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:34.837456 16411 consensus_queue.cc:1035] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:34.848508 16418 mvcc.cc:205] Tried to move back new op lower bound from 7298378116437430272 to 7298378116200153088. Current Snapshot: MvccSnapshot[applied={T|T < 7298378116437430272}]
I20260619 00:56:34.852955 16420 mvcc.cc:205] Tried to move back new op lower bound from 7298378116437430272 to 7298378116200153088. Current Snapshot: MvccSnapshot[applied={T|T < 7298378116437430272}]
I20260619 00:56:34.865163 15927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:34.865509 15927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:34.869494 15927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Sending DeleteTablet for 3 replicas of tablet 97deabca67894d8a9e73a8d1ba7712bf
I20260619 00:56:34.871001 16320 tablet_service.cc:1514] Processing DeleteTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46478
I20260619 00:56:34.871147 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15989
I20260619 00:56:34.871678 16186 tablet_service.cc:1514] Processing DeleteTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43396
I20260619 00:56:34.871767 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 97deabca67894d8a9e73a8d1ba7712bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41736
I20260619 00:56:34.872159 16427 tablet_replica.cc:330] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: stopping tablet replica
I20260619 00:56:34.872756 16427 raft_consensus.cc:2229] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:34.873461 16427 raft_consensus.cc:2258] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:34.875841 16427 ts_tablet_manager.cc:1905] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:34.883307 16427 ts_tablet_manager.cc:1918] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:34.883508 16427 log.cc:1196] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/97deabca67894d8a9e73a8d1ba7712bf
I20260619 00:56:34.884065 16427 ts_tablet_manager.cc:1939] T 97deabca67894d8a9e73a8d1ba7712bf P 542b629aaff64ace8620660481132eda: Deleting consensus metadata
I20260619 00:56:34.886210 15914 catalog_manager.cc:4841] TS 542b629aaff64ace8620660481132eda (127.0.97.3:36777): tablet 97deabca67894d8a9e73a8d1ba7712bf (table test-table [id=324d48ae6e6c4e04908e2738e273eb57]) successfully deleted
I20260619 00:56:34.895891 16428 tablet_replica.cc:330] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: stopping tablet replica
I20260619 00:56:34.896454 16428 raft_consensus.cc:2229] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:56:34.896303 16429 tablet_replica.cc:330] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: stopping tablet replica
I20260619 00:56:34.897234 16428 raft_consensus.cc:2258] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:34.897469 16429 raft_consensus.cc:2229] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:34.898365 16429 raft_consensus.cc:2258] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:34.899595 16428 ts_tablet_manager.cc:1905] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:34.901010 16429 ts_tablet_manager.cc:1905] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 00:56:34 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:56:34.907387 16428 ts_tablet_manager.cc:1918] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:34.907553 16428 log.cc:1196] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/97deabca67894d8a9e73a8d1ba7712bf
I20260619 00:56:34.908334 16428 ts_tablet_manager.cc:1939] T 97deabca67894d8a9e73a8d1ba7712bf P 01edb8fe93604d3788a60834c2a26e4d: Deleting consensus metadata
I20260619 00:56:34.910107 16429 ts_tablet_manager.cc:1918] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:34.910342 16429 log.cc:1196] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/97deabca67894d8a9e73a8d1ba7712bf
I20260619 00:56:34.910498 15913 catalog_manager.cc:4841] TS 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387): tablet 97deabca67894d8a9e73a8d1ba7712bf (table test-table [id=324d48ae6e6c4e04908e2738e273eb57]) successfully deleted
I20260619 00:56:34.910993 16429 ts_tablet_manager.cc:1939] T 97deabca67894d8a9e73a8d1ba7712bf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting consensus metadata
I20260619 00:56:34.913025 15913 catalog_manager.cc:4841] TS bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471): tablet 97deabca67894d8a9e73a8d1ba7712bf (table test-table [id=324d48ae6e6c4e04908e2738e273eb57]) successfully deleted
I20260619 00:56:34.914902 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:34.873310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.97.1:35471)
0619 00:56:34.873582 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:34.873591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:34.892707 (+ 19116us) server_negotiation.cc:407] Connection header received
0619 00:56:34.893062 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:34.893068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:34.893192 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:34.893270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:34.896068 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.897158 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.899765 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:34.900495 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:34.900640 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:34.906713 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:34.906727 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:34.906747 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:34.906809 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:34.910882 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.911962 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.911975 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.911979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.912164 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:34.912698 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:34.912702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:34.912705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:34.913220 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:34.913512 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:34.914132 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 00:56:34.914351 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:56:34.915596 16112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:35.008584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15989
I20260619 00:56:35.028244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16123
Jun 19 00:56:35 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:56:35.048130 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:35.029333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.0.97.2:39387)
0619 00:56:35.029559 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:35.029566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:35.029617 (+ 51us) server_negotiation.cc:407] Connection header received
0619 00:56:35.029687 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:35.029693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:35.029788 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:35.029920 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:35.031378 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:35.032649 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:35.034254 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:35.034989 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:35.035142 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:35.040267 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:35.040292 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:35.040313 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:35.040377 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:35.044558 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:35.045592 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:35.045603 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:35.045608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:35.045807 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:35.046259 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:35.046265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:35.046268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:35.046774 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:35.047161 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:35.047455 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 00:56:35.047705 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:56:35.049058 16246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:35.169538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16123
I20260619 00:56:35.189457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16257
Jun 19 00:56:35 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:56:35.206707 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:35.190202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.97.3:36777)
0619 00:56:35.190409 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:35.190416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:35.190561 (+ 145us) server_negotiation.cc:407] Connection header received
0619 00:56:35.190769 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:35.190775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:35.190883 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:35.190992 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:35.192425 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:35.193270 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:35.194792 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:35.195187 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:35.195331 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:35.200487 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:35.200519 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:35.200537 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:35.200589 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:35.203629 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:35.204623 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:35.204632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:35.204634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:35.204787 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:35.205286 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:35.205289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:35.205291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:35.205675 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:35.205894 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:35.206241 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 00:56:35.206428 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:56:35.207432 16380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:35.300920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16257
I20260619 00:56:35.319692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 15896
I20260619 00:56:35.321009 15958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:35.435043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 15896
I20260619 00:56:55.457976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--webserver_interface=127.0.97.62
--webserver_port=41047
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:56:55.794710 16435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:55.795003 16435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:55.795253 16435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:55.802733 16435 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 00:56:55.802846 16435 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 00:56:55.802935 16435 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 00:56:55.803032 16435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:56:55.803139 16435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:55.803260 16435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:55.803349 16435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:56:55.803429 16435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:56:55.813555 16435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:55.814860 16435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:55.816190 16435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:55.838431 16444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:55.838474 16441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:55.838513 16442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:55.839464 16435 server_base.cc:1034] running on GCE node
I20260619 00:56:55.840374 16435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:55.842007 16435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:55.843271 16435 hybrid_clock.cc:648] HybridClock initialized: now 1781830615843205 us; error 40 us; skew 500 ppm
I20260619 00:56:55.843683 16435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:55 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:55.850941 16435 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:56:55.852658 16435 webserver.cc:460] Webserver started at http://127.0.97.62:41047/ using document root <none> and password file <none>
I20260619 00:56:55.853518 16435 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:55.853668 16435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:55.859728 16435 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 00:56:55.862732 16451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:55.864140 16435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:56:55.864360 16435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "f38e91ccc0974d349658bd98f5b4ec3a"
format_stamp: "Formatted at 2026-06-19 00:56:32 on dist-test-slave-r7bh"
server_key: "461f353b5948c424365048c9ef52becc"
server_key_iv: "07aed39d9cd29a8f8717b3f350e36402"
server_key_version: "encryptionkey@0"
I20260619 00:56:55.865468 16435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:55.889865 16435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:55.894073 16435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:55.894374 16435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:55.913787 16435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37241
I20260619 00:56:55.913806 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37241 every 8 connection(s)
I20260619 00:56:55.915365 16435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:56:55.922256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16435
I20260619 00:56:55.922923 16503 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:56:55.923372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35471
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=36545
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:56:55.928393 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap starting.
I20260619 00:56:55.949956 16503 log.cc:824] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:55.962683 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:56:55.963292 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap complete.
I20260619 00:56:55.973001 16503 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:55.973974 16503 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Initialized, Role: FOLLOWER
I20260619 00:56:55.974640 16503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:55.974898 16503 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:56:55.975023 16503 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:56:55.975222 16503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 1 FOLLOWER]: Advancing to term 2
I20260619 00:56:55.979238 16503 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:55.979837 16503 leader_election.cc:302] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38e91ccc0974d349658bd98f5b4ec3a; no voters:
I20260619 00:56:55.980580 16503 leader_election.cc:288] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 00:56:55.980721 16507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 00:56:55.981997 16507 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 LEADER]: Becoming Leader. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Running, Role: LEADER
I20260619 00:56:55.982683 16507 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:56:55.983036 16503 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:56:55.985289 16508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:56:55.985683 16508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:55.986217 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38e91ccc0974d349658bd98f5b4ec3a. Latest consensus state: current_term: 2 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:56:55.986522 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:56:55.991590 16513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:56:56.002775 16513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:56:56.004374 16513 catalog_manager.cc:1259] Loaded cluster ID: 1b64615b24e24d2ab54ccd8fc9e04c3a
I20260619 00:56:56.004562 16513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:56:56.008844 16513 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:56:56.010568 16513 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Loaded TSK: 0
I20260619 00:56:56.048671 16513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Generated new TSK 1
I20260619 00:56:56.049154 16513 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 00:56:56.280118 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:56.280447 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:56.280609 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:56.288280 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:56.288440 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:56.288616 16505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:56:56.299058 16505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36545
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:56.300416 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:56.301908 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:56.315272 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:56.315749 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:56.315838 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:56.316346 16505 server_base.cc:1034] running on GCE node
I20260619 00:56:56.317042 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:56.318039 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:56.319272 16505 hybrid_clock.cc:648] HybridClock initialized: now 1781830616319227 us; error 52 us; skew 500 ppm
I20260619 00:56:56.319603 16505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:56 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:56.322965 16505 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:56:56.324700 16505 webserver.cc:460] Webserver started at http://127.0.97.1:36545/ using document root <none> and password file <none>
I20260619 00:56:56.325467 16505 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:56.325621 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:56.331517 16505 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 00:56:56.334510 16540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.336027 16505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:56:56.336220 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "db56829b3b68aaa3108d308f2aaca038"
server_key_iv: "a3b9068bbcb9c008d9da4250ed3fda63"
server_key_version: "encryptionkey@0"
I20260619 00:56:56.337388 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:56.352501 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:56.356942 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:56.357235 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:56.358330 16505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:56.359947 16505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:56.360057 16505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.360167 16505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:56.360277 16505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.397083 16505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35471
I20260619 00:56:56.397087 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35471 every 8 connection(s)
I20260619 00:56:56.398813 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:56:56.400389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16505
I20260619 00:56:56.401201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39387
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=45319
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:56:56 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:56.427845 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:56.402294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50641 (local address 127.0.97.62:37241)
0619 00:56:56.402996 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:56.403014 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:56.403835 (+ 821us) server_negotiation.cc:407] Connection header received
0619 00:56:56.407315 (+ 3480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:56.407334 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:56.407775 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:56.408125 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:56.410029 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.412102 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.413927 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.415072 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.415299 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:56.419962 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:56.419971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:56.419987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:56.420027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:56.423181 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:56.424200 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:56.424207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:56.424212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:56.424368 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:56.424920 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:56.424923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:56.424925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:56.425400 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:56.425548 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:56.426578 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0619 00:56:56.426908 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":271,"threads_started":1}
I20260619 00:56:56.428478 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:56.402610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:50641)
0619 00:56:56.403632 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:56.403702 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:56.406633 (+ 2931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:56.408330 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:56.408343 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:56.409206 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:56.409811 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:56.409829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.412300 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.412304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:56.413773 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:56.413786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.415325 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.415330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:56.415531 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:56.416345 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:56.416379 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:56.419615 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:56.423338 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:56.423350 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:56.423370 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:56.424049 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:56.424492 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:56.424499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:56.424506 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:56.424775 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:56.425543 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:56.425556 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:56.426042 (+ 486us) client_negotiation.cc:769] Sending connection context
0619 00:56:56.426404 (+ 362us) client_negotiation.cc:240] Negotiation successful
0619 00:56:56.427045 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":426,"threads_started":1}
I20260619 00:56:56.431389 16653 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:56.431715 16653 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:56.432551 16653 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:56.434787 16468 ts_manager.cc:194] Registered new tserver with Master: bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471)
I20260619 00:56:56.436281 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50641
W20260619 00:56:56.740660 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:56.741006 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:56.741178 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:56.748701 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:56.748878 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:56.749071 16657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:56:56.758889 16657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45319
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:56.760219 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:56.761695 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:56.777362 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:56.777298 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:56.777258 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:56.777972 16657 server_base.cc:1034] running on GCE node
I20260619 00:56:56.778630 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:56.779585 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:56.780857 16657 hybrid_clock.cc:648] HybridClock initialized: now 1781830616780777 us; error 52 us; skew 500 ppm
I20260619 00:56:56.781208 16657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:56 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:56.784556 16657 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:56:56.786125 16657 webserver.cc:460] Webserver started at http://127.0.97.2:45319/ using document root <none> and password file <none>
I20260619 00:56:56.787505 16657 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:56.787711 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:56.793679 16657 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:56:56.796510 16672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.797910 16657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:56:56.798079 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "01edb8fe93604d3788a60834c2a26e4d"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "edcf784baf47652fa0147e27e37df43d"
server_key_iv: "a0684dd9396bf2dae7a8c02b843211b0"
server_key_version: "encryptionkey@0"
I20260619 00:56:56.799077 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:56.824005 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:56.828358 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:56.828631 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:56.829648 16657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:56.830972 16657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:56.831099 16657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.831231 16657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:56.831321 16657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:56.865522 16657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39387
I20260619 00:56:56.865605 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39387 every 8 connection(s)
I20260619 00:56:56.866998 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:56:56.869760 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16657
I20260619 00:56:56.870368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36777
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=34171
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 00:56:56 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:56.896301 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:56.869687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35069 (local address 127.0.97.62:37241)
0619 00:56:56.869917 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:56.869923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:56.878925 (+ 9002us) server_negotiation.cc:407] Connection header received
0619 00:56:56.881087 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:56.881093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:56.881196 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:56.881316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:56.882979 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.883860 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.885739 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.886188 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.886333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:56.890178 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:56.890183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:56.890187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:56.890222 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:56.892738 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:56.893643 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:56.893648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:56.893650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:56.893796 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:56.894345 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:56.894352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:56.894355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:56.894811 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:56.894990 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:56.895705 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 00:56:56.895940 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:56:56.897184 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:56.870274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:35069)
0619 00:56:56.878683 (+ 8409us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:56.878756 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:56.880704 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:56.881590 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:56.881599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:56.882151 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:56.882778 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:56.882797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.884030 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.884040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:56.885581 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:56.885595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:56.886457 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:56.886466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:56.886703 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:56.887481 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:56.887511 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:56.889906 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:56.892916 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:56.892924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:56.892942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:56.893510 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:56.893914 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:56.893920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:56.893924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:56.894174 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:56.894951 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:56.894957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:56.895345 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 00:56:56.895574 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 00:56:56.895963 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8025,"thread_start_us":7813,"threads_started":1}
I20260619 00:56:56.898705 16785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:56.899029 16785 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:56.899755 16785 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:56.901274 16468 ts_manager.cc:194] Registered new tserver with Master: 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387)
I20260619 00:56:56.902200 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35069
W20260619 00:56:57.205950 16788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:56:57.206248 16788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:56:57.206405 16788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:56:57.213778 16788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:56:57.213944 16788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:56:57.214103 16788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:56:57.223723 16788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34171
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:56:57.225077 16788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:56:57.226327 16788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:56:57.239120 16793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:57.239225 16794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:56:57.239259 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:56:57.240362 16788 server_base.cc:1034] running on GCE node
I20260619 00:56:57.241206 16788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:56:57.242331 16788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:56:57.243527 16788 hybrid_clock.cc:648] HybridClock initialized: now 1781830617243462 us; error 58 us; skew 500 ppm
I20260619 00:56:57.243795 16788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:56:57 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:56:57.246930 16788 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:56:57.248472 16788 webserver.cc:460] Webserver started at http://127.0.97.3:34171/ using document root <none> and password file <none>
I20260619 00:56:57.249257 16788 fs_manager.cc:359] Metadata directory not provided
I20260619 00:56:57.249413 16788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:56:57.254994 16788 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 00:56:57.257771 16803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:57.259234 16788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:56:57.259421 16788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "542b629aaff64ace8620660481132eda"
format_stamp: "Formatted at 2026-06-19 00:56:34 on dist-test-slave-r7bh"
server_key: "3acc2316b7d23ef14dc3c030ea609bba"
server_key_iv: "ceb2025bd655ef9a0405f47cb1ea379e"
server_key_version: "encryptionkey@0"
I20260619 00:56:57.260421 16788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:56:57.284560 16788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:56:57.289158 16788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:56:57.289451 16788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:56:57.290515 16788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:56:57.291965 16788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:56:57.292100 16788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:57.292244 16788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:56:57.292333 16788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:56:57.327526 16788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36777
I20260619 00:56:57.327617 16915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36777 every 8 connection(s)
I20260619 00:56:57.329103 16788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:56:57.336424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16788
Jun 19 00:56:57 dist-test-slave-r7bh krb5kdc[15883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:56:57.351092 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.331877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59445 (local address 127.0.97.62:37241)
0619 00:56:57.332089 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.332094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.333310 (+ 1216us) server_negotiation.cc:407] Connection header received
0619 00:56:57.335171 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.335177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.335268 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.335379 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.336929 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.338184 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.339704 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.340137 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.340293 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.343818 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.343825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.343829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.343870 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.347260 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.348283 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.348287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.348289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.348409 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.348925 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.348930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.348933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.349278 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.349401 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.350399 (+ 998us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.350665 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":7040}
I20260619 00:56:57.352188 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.332380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:59445)
0619 00:56:57.333158 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.333195 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.334900 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.335629 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.335637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.336135 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:56:57.336745 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.336764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.338356 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.338359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.339556 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.339567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.340385 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.340392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.340653 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.341395 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:56:57.341428 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:56:57.343605 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:56:57.347411 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:57.347420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:57.347438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:57.348146 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:57.348521 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:56:57.348528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:56:57.348533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:56:57.348768 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:56:57.349418 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:56:57.349429 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:56:57.349907 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.350230 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.350762 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":307,"threads_started":1}
I20260619 00:56:57.353670 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.337416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.97.62:37241)
0619 00:56:57.337935 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.337939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.338370 (+ 431us) server_negotiation.cc:407] Connection header received
0619 00:56:57.338575 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.338586 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.338650 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.338758 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.340280 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.341131 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.342697 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.343176 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.343298 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.346953 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.346957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.346961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.347008 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.350413 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.351490 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.351495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.351498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.351626 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.352000 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.352005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.352007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.352360 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.352616 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.353015 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.353265 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":896,"thread_start_us":294,"threads_started":1}
I20260619 00:56:57.354130 16916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37241
I20260619 00:56:57.354460 16916 heartbeater.cc:464] Registering TS with master...
I20260619 00:56:57.355279 16916 heartbeater.cc:511] Master 127.0.97.62:37241 requested a full tablet report, sending...
I20260619 00:56:57.356746 16468 ts_manager.cc:194] Registered new tserver with Master: 542b629aaff64ace8620660481132eda (127.0.97.3:36777)
I20260619 00:56:57.357595 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59445
I20260619 00:56:57.368733 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:56:57.375550 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.369720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.97.62:37241)
0619 00:56:57.369909 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.369913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.370413 (+ 500us) server_negotiation.cc:407] Connection header received
0619 00:56:57.370599 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.370609 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.370764 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.370886 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:57.371277 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.372089 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.373833 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.374245 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.374336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.374813 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:57.375027 (+ 214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 00:56:57.375315 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60550: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 00:56:57.375645 16919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60550: token expired
W20260619 00:56:57.376202 16390 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:37241)
I20260619 00:56:57.389518 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.376717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.97.62:37241)
0619 00:56:57.376903 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.376907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.376938 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:56:57.377072 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.377077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.377141 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.377212 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.378493 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.379183 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.380459 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.380942 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.381041 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.383904 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.383909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.383914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.383990 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.386723 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.387574 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.387578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.387580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.387731 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.388124 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.388129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.388131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.388472 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.388691 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.389051 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.389238 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 00:56:57.393141 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:56:57.401113 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:56:57.424875 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.413866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40896)
0619 00:56:57.415835 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.415882 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.416087 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.417243 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.417252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.417284 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:57.417743 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.417758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.419517 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.419524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.422025 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.422044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.423856 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.423862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.424062 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.424135 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.424280 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.424488 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"thread_start_us":1588,"threads_started":1}
I20260619 00:56:57.425658 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.414076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.0.97.1:35471)
0619 00:56:57.415489 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.415498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.416304 (+ 806us) server_negotiation.cc:407] Connection header received
0619 00:56:57.416416 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.416426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.416718 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.416953 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:57.417964 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.419380 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.422211 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.423648 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.423802 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.424063 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.424325 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.424497 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":1062,"threads_started":1}
I20260619 00:56:57.426034 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.416362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39387 (local address 127.0.0.1:51166)
0619 00:56:57.417324 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.417352 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.417524 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.418726 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.418731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.418756 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:57.419223 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.419240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.421518 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.421524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.423250 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.423261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.425142 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.425148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.425338 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.425368 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.425438 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.425587 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":48,"spinlock_wait_cycles":2816,"thread_start_us":739,"threads_started":1}
I20260619 00:56:57.426483 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.414012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36777 (local address 127.0.0.1:43262)
0619 00:56:57.418050 (+ 4038us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.418068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.418186 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.418889 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.418894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.418920 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:57.419341 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.419352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.420842 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.420850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.423223 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.423238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.425643 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.425649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.425827 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.425865 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.425945 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.426103 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2835,"mutex_wait_us":5,"thread_start_us":2664,"threads_started":1}
I20260619 00:56:57.426815 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.414855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.97.3:36777)
0619 00:56:57.416521 (+ 1666us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.416529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.418179 (+ 1650us) server_negotiation.cc:407] Connection header received
0619 00:56:57.418348 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.418356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.418596 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.418786 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:57.419534 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.420645 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.423397 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.424880 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.425017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.426291 (+ 1274us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.426366 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.426526 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495,"thread_start_us":1319,"threads_started":1}
I20260619 00:56:57.427241 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.416914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.0.97.2:39387)
0619 00:56:57.418120 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.418130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.418158 (+ 28us) server_negotiation.cc:407] Connection header received
0619 00:56:57.418252 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.418261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.418503 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.419447 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:57.419707 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.421343 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.423452 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.424941 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.426367 (+ 1426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.426613 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.426677 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.426861 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":870,"threads_started":1}
I20260619 00:56:57.429988 16588 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf (DEFAULT_TABLE table=test-table [id=b1b19df251a244ecaa30966e30ac277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:57.430143 16720 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf (DEFAULT_TABLE table=test-table [id=b1b19df251a244ecaa30966e30ac277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:57.430425 16851 tablet_service.cc:1467] Processing CreateTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf (DEFAULT_TABLE table=test-table [id=b1b19df251a244ecaa30966e30ac277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:56:57.431461 16720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fbc0ab6a54558a86a3a71867b7cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:57.431566 16851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fbc0ab6a54558a86a3a71867b7cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:57.431566 16588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7fbc0ab6a54558a86a3a71867b7cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:56:57.439877 16653 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:57.445112 16932 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Bootstrap starting.
I20260619 00:56:57.447459 16933 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Bootstrap starting.
I20260619 00:56:57.448328 16934 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Bootstrap starting.
I20260619 00:56:57.449205 16932 tablet_bootstrap.cc:654] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:57.450614 16932 log.cc:824] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:57.452324 16933 tablet_bootstrap.cc:654] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:57.452349 16934 tablet_bootstrap.cc:654] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Neither blocks nor log segments found. Creating new log.
I20260619 00:56:57.453708 16933 log.cc:824] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:57.453708 16934 log.cc:824] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Log is configured to *not* fsync() on all Append() calls
I20260619 00:56:57.454146 16932 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: No bootstrap required, opened a new log
I20260619 00:56:57.454561 16932 ts_tablet_manager.cc:1397] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 00:56:57.456539 16934 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: No bootstrap required, opened a new log
I20260619 00:56:57.456869 16934 ts_tablet_manager.cc:1397] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 00:56:57.456940 16933 tablet_bootstrap.cc:492] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: No bootstrap required, opened a new log
I20260619 00:56:57.457206 16933 ts_tablet_manager.cc:1397] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260619 00:56:57.465009 16934 raft_consensus.cc:348] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.465580 16934 raft_consensus.cc:374] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:57.465739 16934 raft_consensus.cc:729] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542b629aaff64ace8620660481132eda, State: Initialized, Role: FOLLOWER
I20260619 00:56:57.465567 16932 raft_consensus.cc:348] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.466033 16932 raft_consensus.cc:374] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:57.466183 16932 raft_consensus.cc:729] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01edb8fe93604d3788a60834c2a26e4d, State: Initialized, Role: FOLLOWER
I20260619 00:56:57.466506 16934 consensus_queue.cc:260] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.466820 16932 consensus_queue.cc:260] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.467744 16916 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:57.468465 16934 ts_tablet_manager.cc:1428] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 00:56:57.468946 16933 raft_consensus.cc:348] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.469462 16933 raft_consensus.cc:374] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:56:57.469587 16785 heartbeater.cc:503] Master 127.0.97.62:37241 was elected leader, sending a full tablet report...
I20260619 00:56:57.469604 16933 raft_consensus.cc:729] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe6bc1fd98f4ba1b0a906eae9edfbcc, State: Initialized, Role: FOLLOWER
I20260619 00:56:57.470376 16932 ts_tablet_manager.cc:1428] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260619 00:56:57.470268 16933 consensus_queue.cc:260] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.472298 16933 ts_tablet_manager.cc:1428] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 00:56:57.493582 16940 raft_consensus.cc:482] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:56:57.494052 16940 raft_consensus.cc:504] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.496083 16940 leader_election.cc:288] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 542b629aaff64ace8620660481132eda (127.0.97.3:36777), 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387)
I20260619 00:56:57.502823 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.496397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55611 (local address 127.0.97.3:36777)
0619 00:56:57.496594 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.496600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.497404 (+ 804us) server_negotiation.cc:407] Connection header received
0619 00:56:57.497635 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.497644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.497707 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.497802 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:57.498409 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.499128 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.501040 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.501834 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.501982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.502109 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.502420 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.502549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 00:56:57.502954 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.496616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58079 (local address 127.0.97.2:39387)
0619 00:56:57.496882 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.496892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.497584 (+ 692us) server_negotiation.cc:407] Connection header received
0619 00:56:57.497781 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.497786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.497843 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.497939 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:56:57.498551 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.499236 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.501075 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.501965 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.502117 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.502224 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.502515 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.502685 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:56:57.503006 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.496394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36777 (local address 127.0.97.1:55611)
0619 00:56:57.497284 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.497301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.497477 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.497817 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.497822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.497845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:57.498253 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.498263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.499270 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.499276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.500872 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.500884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.502061 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.502073 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.502250 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.502273 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.502386 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.502518 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":20,"spinlock_wait_cycles":33664,"thread_start_us":503,"threads_started":1}
I20260619 00:56:57.503328 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.496400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39387 (local address 127.0.97.1:58079)
0619 00:56:57.497474 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.497489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:56:57.497633 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:56:57.497963 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:56:57.497967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:56:57.498005 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:56:57.498399 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.498407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.499364 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.499374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.500922 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:56:57.500939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.502192 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.502197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:56:57.502365 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.502388 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:56:57.502489 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 00:56:57.502634 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":251,"thread_start_us":565,"threads_started":1}
I20260619 00:56:57.504276 16871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fbc0ab6a54558a86a3a71867b7cdf" candidate_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542b629aaff64ace8620660481132eda" is_pre_election: true
I20260619 00:56:57.504292 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fbc0ab6a54558a86a3a71867b7cdf" candidate_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01edb8fe93604d3788a60834c2a26e4d" is_pre_election: true
I20260619 00:56:57.504760 16871 raft_consensus.cc:2454] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe6bc1fd98f4ba1b0a906eae9edfbcc in term 0.
I20260619 00:56:57.504768 16740 raft_consensus.cc:2454] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe6bc1fd98f4ba1b0a906eae9edfbcc in term 0.
I20260619 00:56:57.505509 16543 leader_election.cc:302] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01edb8fe93604d3788a60834c2a26e4d, 542b629aaff64ace8620660481132eda, bfe6bc1fd98f4ba1b0a906eae9edfbcc; no voters:
I20260619 00:56:57.505908 16940 raft_consensus.cc:2781] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:56:57.506064 16940 raft_consensus.cc:482] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:56:57.506186 16940 raft_consensus.cc:3037] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:57.508378 16940 raft_consensus.cc:504] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.509117 16940 leader_election.cc:288] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [CANDIDATE]: Term 1 election: Requested vote from peers 542b629aaff64ace8620660481132eda (127.0.97.3:36777), 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387)
I20260619 00:56:57.509555 16871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fbc0ab6a54558a86a3a71867b7cdf" candidate_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542b629aaff64ace8620660481132eda"
I20260619 00:56:57.509598 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7fbc0ab6a54558a86a3a71867b7cdf" candidate_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01edb8fe93604d3788a60834c2a26e4d"
I20260619 00:56:57.509759 16871 raft_consensus.cc:3037] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:57.509833 16740 raft_consensus.cc:3037] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:56:57.512032 16871 raft_consensus.cc:2454] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe6bc1fd98f4ba1b0a906eae9edfbcc in term 1.
I20260619 00:56:57.512322 16740 raft_consensus.cc:2454] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe6bc1fd98f4ba1b0a906eae9edfbcc in term 1.
I20260619 00:56:57.512442 16543 leader_election.cc:302] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542b629aaff64ace8620660481132eda, bfe6bc1fd98f4ba1b0a906eae9edfbcc; no voters:
I20260619 00:56:57.512734 16940 raft_consensus.cc:2781] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:56:57.513274 16940 raft_consensus.cc:686] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 LEADER]: Becoming Leader. State: Replica: bfe6bc1fd98f4ba1b0a906eae9edfbcc, State: Running, Role: LEADER
I20260619 00:56:57.513762 16940 consensus_queue.cc:237] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } }
I20260619 00:56:57.518628 16466 catalog_manager.cc:5495] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260619 00:56:57.551586 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.545896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.97.1:35471)
0619 00:56:57.546085 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.546090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.546107 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:56:57.546198 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.546201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.546292 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.546450 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:56:57.546793 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.547542 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.549475 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.549987 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.550093 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.550594 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:56:57.551115 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:56:57.551233 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.551303 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.551376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:56:57.560130 16871 raft_consensus.cc:1261] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Refusing update from remote peer bfe6bc1fd98f4ba1b0a906eae9edfbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:57.560163 16740 raft_consensus.cc:1261] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Refusing update from remote peer bfe6bc1fd98f4ba1b0a906eae9edfbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:56:57.561312 16940 consensus_queue.cc:1035] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "01edb8fe93604d3788a60834c2a26e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:57.561676 16943 consensus_queue.cc:1035] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "542b629aaff64ace8620660481132eda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:56:57.575197 16946 mvcc.cc:205] Tried to move back new op lower bound from 7298378209516019712 to 7298378209340854272. Current Snapshot: MvccSnapshot[applied={T|T < 7298378209516019712}]
I20260619 00:56:57.589347 16466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:57.589674 16466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:56:57.593362 16466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Sending DeleteTablet for 3 replicas of tablet cc7fbc0ab6a54558a86a3a71867b7cdf
I20260619 00:56:57.594558 16851 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43262
I20260619 00:56:57.594828 16720 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51166
I20260619 00:56:57.594991 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16505
I20260619 00:56:57.595577 16955 tablet_replica.cc:330] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: stopping tablet replica
I20260619 00:56:57.595726 16588 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7fbc0ab6a54558a86a3a71867b7cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:56:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40896
I20260619 00:56:57.596565 16956 tablet_replica.cc:330] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: stopping tablet replica
I20260619 00:56:57.597039 16956 raft_consensus.cc:2229] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:57.597555 16956 raft_consensus.cc:2258] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:57.598312 16955 raft_consensus.cc:2229] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:56:57.600266 16956 ts_tablet_manager.cc:1905] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:57.600838 16955 raft_consensus.cc:2258] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:57.603919 16955 ts_tablet_manager.cc:1905] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:57.608306 16957 tablet_replica.cc:330] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: stopping tablet replica
I20260619 00:56:57.608791 16957 raft_consensus.cc:2229] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:56:57.609572 16957 raft_consensus.cc:2258] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:56:57.613806 16955 ts_tablet_manager.cc:1918] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:57.613976 16955 log.cc:1196] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/cc7fbc0ab6a54558a86a3a71867b7cdf
I20260619 00:56:57.614465 16956 ts_tablet_manager.cc:1918] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:56:57.614734 16956 log.cc:1196] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/cc7fbc0ab6a54558a86a3a71867b7cdf
I20260619 00:56:57.615324 16955 ts_tablet_manager.cc:1939] T cc7fbc0ab6a54558a86a3a71867b7cdf P 542b629aaff64ace8620660481132eda: Deleting consensus metadata
I20260619 00:56:57.615353 16956 ts_tablet_manager.cc:1939] T cc7fbc0ab6a54558a86a3a71867b7cdf P 01edb8fe93604d3788a60834c2a26e4d: Deleting consensus metadata
I20260619 00:56:57.615648 16957 ts_tablet_manager.cc:1905] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:56:57.617250 16454 catalog_manager.cc:4841] TS 542b629aaff64ace8620660481132eda (127.0.97.3:36777): tablet cc7fbc0ab6a54558a86a3a71867b7cdf (table test-table [id=b1b19df251a244ecaa30966e30ac277a]) successfully deleted
I20260619 00:56:57.617457 16453 catalog_manager.cc:4841] TS 01edb8fe93604d3788a60834c2a26e4d (127.0.97.2:39387): tablet cc7fbc0ab6a54558a86a3a71867b7cdf (table test-table [id=b1b19df251a244ecaa30966e30ac277a]) successfully deleted
I20260619 00:56:57.619679 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.596017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.0.97.1:35471)
0619 00:56:57.596745 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.596753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.596775 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:56:57.596864 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.596873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.596983 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.597111 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.599664 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.601315 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.604004 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.605025 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.605227 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.609447 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.609460 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.609476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.609529 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.614080 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.616837 (+ 2757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.616846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.616849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.617055 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.617927 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.617934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.617942 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.618289 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.618692 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.619062 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.619295 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554}
I20260619 00:56:57.620426 16648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:57.719537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16505
I20260619 00:56:57.734731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16657
W20260619 00:56:57.734757 16453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35471 (error 108)
I20260619 00:56:57.736867 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.736414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40912)
0619 00:56:57.736577 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.736675 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 00:56:57.737267 16453 catalog_manager.cc:4584] TS bfe6bc1fd98f4ba1b0a906eae9edfbcc (127.0.97.1:35471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc7fbc0ab6a54558a86a3a71867b7cdf: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
I20260619 00:56:57.752348 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.735548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.0.97.2:39387)
0619 00:56:57.735750 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.735759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.735840 (+ 81us) server_negotiation.cc:407] Connection header received
0619 00:56:57.736001 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.736007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.736099 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.736236 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.737938 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.738677 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.739959 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.740338 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.740433 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.746888 (+ 6455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.746904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.746921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.746982 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.749882 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.750566 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.750573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.750575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.750694 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.751008 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.751011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.751013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.751379 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.751601 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.751820 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.751986 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:56:57.753247 16780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:57.791167 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.790674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40926)
0619 00:56:57.790895 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.790997 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 00:56:57.792294 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.791953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40928)
0619 00:56:57.792098 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.792184 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 00:56:57.842339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16657
I20260619 00:56:57.852768 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.852333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40934)
0619 00:56:57.852527 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.852623 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 00:56:57.853927 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.853586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40936)
0619 00:56:57.853741 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.853822 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:56:57.860805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16788
I20260619 00:56:57.874818 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.861630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.0.97.3:36777)
0619 00:56:57.861810 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 00:56:57.861815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:56:57.861833 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:56:57.861973 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:56:57.861977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:56:57.862044 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:56:57.862126 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:56:57.863550 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.864351 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.865827 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:56:57.866227 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:56:57.866324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:56:57.869283 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:56:57.869297 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:56:57.869314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:56:57.869361 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:56:57.872099 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.872762 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.872772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.872775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.873082 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:56:57.873461 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:56:57.873464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:56:57.873466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:56:57.873803 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:56:57.874012 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 00:56:57.874279 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 00:56:57.874464 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:56:57.875533 16911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:57.935101 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.934578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40946)
0619 00:56:57.934822 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.934924 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 00:56:57.936269 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:56:57.935880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35471 (local address 127.0.0.1:40960)
0619 00:56:57.936089 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 00:56:57.936169 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 00:56:57.966367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16788
I20260619 00:56:57.984004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 16435
I20260619 00:56:57.985327 16498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:56:58.104923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 16435
I20260619 00:57:18.127270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--webserver_interface=127.0.97.62
--webserver_port=41047
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:18.466032 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:18.466358 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:18.466511 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:18.474251 16963 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 00:57:18.474364 16963 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 00:57:18.474443 16963 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 00:57:18.474538 16963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:18.474629 16963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:18.474735 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:18.474809 16963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:18.474895 16963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:18.485334 16963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:18.486795 16963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:18.488214 16963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:18.500576 16969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:18.500541 16968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:18.500541 16971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:18.501129 16963 server_base.cc:1034] running on GCE node
I20260619 00:57:18.502200 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:18.503756 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:18.505008 16963 hybrid_clock.cc:648] HybridClock initialized: now 1781830638504938 us; error 49 us; skew 500 ppm
I20260619 00:57:18.505373 16963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:18 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:18.513808 16963 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:57:18.515434 16963 webserver.cc:460] Webserver started at http://127.0.97.62:41047/ using document root <none> and password file <none>
I20260619 00:57:18.516224 16963 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:18.516373 16963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:18.522291 16963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:57:18.525513 16978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:18.526999 16963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:18.527155 16963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "f38e91ccc0974d349658bd98f5b4ec3a"
format_stamp: "Formatted at 2026-06-19 00:56:32 on dist-test-slave-r7bh"
server_key: "461f353b5948c424365048c9ef52becc"
server_key_iv: "07aed39d9cd29a8f8717b3f350e36402"
server_key_version: "encryptionkey@0"
I20260619 00:57:18.528137 16963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:18.563807 16963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:18.568614 16963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:18.568898 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:18.588312 16963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37241
I20260619 00:57:18.588357 17029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37241 every 8 connection(s)
I20260619 00:57:18.589982 16963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:18.591647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 16963
I20260619 00:57:18.592666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35471
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=36545
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:57:18.600405 17030 sys_catalog.cc:263] Verifying existing consensus state
I20260619 00:57:18.604303 17030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap starting.
I20260619 00:57:18.625752 17030 log.cc:824] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:18.641935 17030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 00:57:18.642475 17030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Bootstrap complete.
I20260619 00:57:18.651381 17030 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:57:18.652348 17030 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Initialized, Role: FOLLOWER
I20260619 00:57:18.653103 17030 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:57:18.653375 17030 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:18.653489 17030 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:18.653684 17030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 2 FOLLOWER]: Advancing to term 3
I20260619 00:57:18.657433 17030 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:57:18.657927 17030 leader_election.cc:302] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38e91ccc0974d349658bd98f5b4ec3a; no voters:
I20260619 00:57:18.658591 17030 leader_election.cc:288] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 00:57:18.658751 17034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 3 FOLLOWER]: Leader election won for term 3
I20260619 00:57:18.659759 17034 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [term 3 LEADER]: Becoming Leader. State: Replica: f38e91ccc0974d349658bd98f5b4ec3a, State: Running, Role: LEADER
I20260619 00:57:18.660364 17034 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } }
I20260619 00:57:18.660756 17030 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:18.663168 17036 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38e91ccc0974d349658bd98f5b4ec3a. Latest consensus state: current_term: 3 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:57:18.663563 17036 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:18.664079 17035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38e91ccc0974d349658bd98f5b4ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37241 } } }
I20260619 00:57:18.664425 17035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:18.669919 17040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:18.680507 17040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:18.682731 17040 catalog_manager.cc:1259] Loaded cluster ID: 1b64615b24e24d2ab54ccd8fc9e04c3a
I20260619 00:57:18.682938 17040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:18.688423 17040 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:18.690649 17040 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Loaded TSK: 0
I20260619 00:57:18.690938 17040 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f38e91ccc0974d349658bd98f5b4ec3a: Loaded TSK: 1
F20260619 00:57:18.692557 17040 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03077ed9ad google::LogMessage::Fail() at ??:0
@ 0x7f03077f088a google::LogMessage::SendToLog() at ??:0
@ 0x7f03077ed4de google::LogMessage::Flush() at ??:0
@ 0x7f03077eea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f030b5517d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f030b54073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f031d21c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f031d21b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f031d2a72a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f031d2a70c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f031d2f15e9 std::function<>::operator()() at ??:0
@ 0x7f031d21fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f031d21dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f031d2a5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f031d2a5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f030a439345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f030a43c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f030a43c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f030a417eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f030d6f16db start_thread at ??:0
@ 0x7f030668761f clone at ??:0
W20260619 00:57:18.990751 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:18.991093 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:18.991264 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:18.998839 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:18.999001 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:18.999186 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:19.009667 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36545
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:19.011039 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:19.012414 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:19.026546 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.026578 17080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.026503 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:19.027580 17032 server_base.cc:1034] running on GCE node
I20260619 00:57:19.028237 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:19.029378 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:19.030587 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781830639030536 us; error 49 us; skew 500 ppm
I20260619 00:57:19.030965 17032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:19 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:19.034307 17032 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:57:19.035717 17032 webserver.cc:460] Webserver started at http://127.0.97.1:36545/ using document root <none> and password file <none>
I20260619 00:57:19.036378 17032 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:19.036479 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:19.041899 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:19.044798 17087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:19.046257 17032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:57:19.046443 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "bfe6bc1fd98f4ba1b0a906eae9edfbcc"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "db56829b3b68aaa3108d308f2aaca038"
server_key_iv: "a3b9068bbcb9c008d9da4250ed3fda63"
server_key_version: "encryptionkey@0"
I20260619 00:57:19.047441 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:19.082700 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:19.087064 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:19.087322 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:19.088326 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:19.090484 17094 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 00:57:19.096164 17094 ts_tablet_manager.cc:1852] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 00:57:19.097270 17094 ts_tablet_manager.cc:1905] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:19.104359 17094 ts_tablet_manager.cc:1918] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:19.104596 17094 log.cc:1196] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/cc7fbc0ab6a54558a86a3a71867b7cdf
I20260619 00:57:19.105139 17094 ts_tablet_manager.cc:1939] T cc7fbc0ab6a54558a86a3a71867b7cdf P bfe6bc1fd98f4ba1b0a906eae9edfbcc: Deleting consensus metadata
I20260619 00:57:19.105535 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 00:57:19.105652 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260619 00:57:19.105758 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:19.105822 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:19.140467 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35471
I20260619 00:57:19.140556 17200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35471 every 8 connection(s)
I20260619 00:57:19.141942 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:19.146998 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.145139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:51649)
0619 00:57:19.145912 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.146107 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":299,"threads_started":1}
W20260619 00:57:19.147655 17201 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37241 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37241: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
I20260619 00:57:19.148469 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.148171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:50471)
0619 00:57:19.148302 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.148369 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 00:57:19.149339 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.149016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:48379)
0619 00:57:19.149152 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.149229 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 00:57:19.149561 17201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 00:57:19.151446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17032
I20260619 00:57:19.152168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39387
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=45319
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:19.489058 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:19.489377 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:19.489552 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:19.497057 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:19.497201 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:19.497391 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:19.507499 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45319
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:19.508965 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:19.510344 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:19.523454 17209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.523427 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.523672 17212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:19.524230 17204 server_base.cc:1034] running on GCE node
I20260619 00:57:19.524958 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:19.526064 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:19.527287 17204 hybrid_clock.cc:648] HybridClock initialized: now 1781830639527238 us; error 64 us; skew 500 ppm
I20260619 00:57:19.527624 17204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:19 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:19.530884 17204 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:57:19.532382 17204 webserver.cc:460] Webserver started at http://127.0.97.2:45319/ using document root <none> and password file <none>
I20260619 00:57:19.533169 17204 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:19.533324 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:19.538604 17204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 00:57:19.541366 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:19.542764 17204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:57:19.542948 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "01edb8fe93604d3788a60834c2a26e4d"
format_stamp: "Formatted at 2026-06-19 00:56:33 on dist-test-slave-r7bh"
server_key: "edcf784baf47652fa0147e27e37df43d"
server_key_iv: "a0684dd9396bf2dae7a8c02b843211b0"
server_key_version: "encryptionkey@0"
I20260619 00:57:19.543989 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:19.573244 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:19.577836 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:19.578071 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:19.579041 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:19.580397 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:19.580495 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:19.580565 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:19.580621 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:19.614684 17204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39387
I20260619 00:57:19.614799 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39387 every 8 connection(s)
I20260619 00:57:19.616254 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:19.621634 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.619380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:57867)
0619 00:57:19.620272 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.620546 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":321,"threads_started":1}
W20260619 00:57:19.622370 17332 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37241 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37241: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
I20260619 00:57:19.623216 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.622863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:55717)
0619 00:57:19.623025 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.623100 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 00:57:19.624174 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:19.623798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:48845)
0619 00:57:19.623960 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:19.624054 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 00:57:19.624392 17332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 00:57:19.625932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17204
I20260619 00:57:19.626581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36777
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37241
--webserver_port=34171
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:19.960667 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:19.961089 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:19.961280 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:19.968776 17335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:19.968988 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:19.969185 17335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:19.979272 17335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34171
--tserver_master_addrs=127.0.97.62:37241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:19.980659 17335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:19.982041 17335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:19.995544 17343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.995566 17341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:19.995857 17340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:19.996358 17335 server_base.cc:1034] running on GCE node
I20260619 00:57:19.997053 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:19.998076 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:19.999293 17335 hybrid_clock.cc:648] HybridClock initialized: now 1781830639999245 us; error 65 us; skew 500 ppm
I20260619 00:57:19.999635 17335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:19 dist-test-slave-r7bh krb5kdc[15883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:20.003067 17335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:57:20.004628 17335 webserver.cc:460] Webserver started at http://127.0.97.3:34171/ using document root <none> and password file <none>
I20260619 00:57:20.005412 17335 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:20.005563 17335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:20.011415 17335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 00:57:20.014269 17350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:20.015645 17335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:57:20.015830 17335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "542b629aaff64ace8620660481132eda"
format_stamp: "Formatted at 2026-06-19 00:56:34 on dist-test-slave-r7bh"
server_key: "3acc2316b7d23ef14dc3c030ea609bba"
server_key_iv: "ceb2025bd655ef9a0405f47cb1ea379e"
server_key_version: "encryptionkey@0"
I20260619 00:57:20.016889 17335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:20.034296 17335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:20.039147 17335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:20.039444 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:20.040524 17335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:20.042085 17335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:20.042220 17335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:20.042363 17335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:20.042459 17335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:20.077423 17335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36777
I20260619 00:57:20.077515 17462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36777 every 8 connection(s)
I20260619 00:57:20.078902 17335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:20.079914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17335
I20260619 00:57:20.084049 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:20.082270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:46833)
0619 00:57:20.082982 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:20.083169 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
W20260619 00:57:20.084779 17463 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37241 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37241: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
I20260619 00:57:20.085690 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:20.085346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:33891)
0619 00:57:20.085507 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:20.085577 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 00:57:20.086634 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:20.086334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:46407)
0619 00:57:20.086453 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:20.086526 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 00:57:20.086979 17463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 00:57:20.150924 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:20.150103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:34473)
0619 00:57:20.150586 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:20.150682 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":210,"threads_started":1}
I20260619 00:57:20.625793 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:20.624997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:58267)
0619 00:57:20.625444 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:20.625566 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260619 00:57:21.088313 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:21.087514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:40499)
0619 00:57:21.087964 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:21.088049 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260619 00:57:21.152755 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:21.151908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:38505)
0619 00:57:21.152412 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:21.152515 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
I20260619 00:57:21.627388 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:21.626687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:35913)
0619 00:57:21.627139 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:21.627222 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":215,"threads_started":1}
I20260619 00:57:22.090086 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:22.089296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:52393)
0619 00:57:22.089817 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:22.089918 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
I20260619 00:57:22.154433 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:22.153732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:53721)
0619 00:57:22.154177 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:22.154258 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260619 00:57:22.629158 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:22.628342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:34809)
0619 00:57:22.628888 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:22.628985 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":303,"threads_started":1}
I20260619 00:57:23.091804 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:23.091061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:38325)
0619 00:57:23.091548 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:23.091632 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260619 00:57:23.156018 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:23.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:53683)
0619 00:57:23.155777 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:23.155858 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260619 00:57:23.630728 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:23.630007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:40817)
0619 00:57:23.630477 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:23.630555 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260619 00:57:24.093618 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:24.092803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:52393)
0619 00:57:24.093340 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:24.093421 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":218,"threads_started":1}
I20260619 00:57:24.157608 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:24.156941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:45991)
0619 00:57:24.157355 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:24.157435 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260619 00:57:24.632535 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:24.631764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:44397)
0619 00:57:24.632277 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:24.632364 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260619 00:57:25.095463 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:25.094623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:58235)
0619 00:57:25.095176 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:25.095271 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260619 00:57:25.159174 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:25.158476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:43581)
0619 00:57:25.158927 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:25.159010 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260619 00:57:25.634272 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:25.633546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:53241)
0619 00:57:25.634016 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:25.634107 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260619 00:57:26.097193 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:26.096350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:56877)
0619 00:57:26.096945 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:26.097033 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":239,"threads_started":1}
I20260619 00:57:26.161124 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:26.160194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:42099)
0619 00:57:26.160756 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:26.160916 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":263,"threads_started":1}
I20260619 00:57:26.636232 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:26.635327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:33169)
0619 00:57:26.635915 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:26.636012 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":259,"threads_started":1}
I20260619 00:57:27.099210 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:27.098385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:40317)
0619 00:57:27.098913 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:27.098994 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":241,"threads_started":1}
I20260619 00:57:27.162855 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:27.162072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:55647)
0619 00:57:27.162570 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:27.162685 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":275,"threads_started":1}
I20260619 00:57:27.638008 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:27.637255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:50253)
0619 00:57:27.637764 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:27.637845 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":259,"threads_started":1}
I20260619 00:57:28.101001 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:28.100201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:53743)
0619 00:57:28.100673 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:28.100755 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260619 00:57:28.164517 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:28.163744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:41501)
0619 00:57:28.164245 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:28.164347 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
I20260619 00:57:28.639716 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:28.638970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:48897)
0619 00:57:28.639448 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:28.639535 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260619 00:57:29.102694 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:29.101885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:58879)
0619 00:57:29.102415 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:29.102500 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260619 00:57:29.166391 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:29.165663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:57647)
0619 00:57:29.166119 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:29.166208 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
I20260619 00:57:29.641438 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:29.640692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:47167)
0619 00:57:29.641193 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:29.641273 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260619 00:57:30.104362 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:30.103659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:41543)
0619 00:57:30.104120 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:30.104204 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":241,"threads_started":1}
I20260619 00:57:30.168156 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:30.167428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:37223)
0619 00:57:30.167875 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:30.167953 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":255,"threads_started":1}
I20260619 00:57:30.643149 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:30.642372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:45029)
0619 00:57:30.642840 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:30.642959 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260619 00:57:31.106233 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:31.105472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:38873)
0619 00:57:31.105967 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:31.106059 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260619 00:57:31.169893 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:31.169179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:37351)
0619 00:57:31.169643 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:31.169725 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260619 00:57:31.645112 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:31.644309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:58045)
0619 00:57:31.644777 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:31.644944 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":241,"threads_started":1}
I20260619 00:57:32.107860 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:32.107134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:34027)
0619 00:57:32.107616 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:32.107694 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":229,"threads_started":1}
I20260619 00:57:32.171653 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:32.170864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:37297)
0619 00:57:32.171340 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:32.171434 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260619 00:57:32.646827 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:32.646060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:53693)
0619 00:57:32.646535 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:32.646634 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":224,"threads_started":1}
I20260619 00:57:33.109620 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:33.108872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:58465)
0619 00:57:33.109338 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:33.109421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":231,"threads_started":1}
I20260619 00:57:33.173501 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:33.172589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:37101)
0619 00:57:33.173173 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:33.173258 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":331,"threads_started":1}
I20260619 00:57:33.648869 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:33.647988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:38833)
0619 00:57:33.648484 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:33.648575 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":241,"threads_started":1}
I20260619 00:57:34.111480 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:34.110711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:48593)
0619 00:57:34.111197 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:34.111284 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":256,"threads_started":1}
I20260619 00:57:34.175293 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:34.174565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:58913)
0619 00:57:34.175021 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:34.175109 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260619 00:57:34.651077 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:34.650134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.2:34329)
0619 00:57:34.650719 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:34.650839 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":329,"threads_started":1}
I20260619 00:57:35.088442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17032
I20260619 00:57:35.107967 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:35.089732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.0.97.1:35471)
0619 00:57:35.090281 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:35.090300 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:35.090349 (+ 49us) server_negotiation.cc:407] Connection header received
0619 00:57:35.092583 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:35.092595 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:35.093136 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:35.093435 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:35.094969 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.096242 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.097709 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.098393 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.098530 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:35.101598 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:35.101605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:35.101622 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:35.101665 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:35.104675 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.105621 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.105627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.105634 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.105796 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.106150 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.106153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.106155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.106678 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:35.106911 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:35.107338 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 00:57:35.107670 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":281,"threads_started":1}
I20260619 00:57:35.109544 17196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:35.113875 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:35.112857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.3:36879)
0619 00:57:35.113605 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:35.113703 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":497,"threads_started":1}
I20260619 00:57:35.194684 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:35.193612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37241 (local address 127.0.97.1:41005)
0619 00:57:35.194168 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:35.194370 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
I20260619 00:57:35.194731 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17032
I20260619 00:57:35.212033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17204
I20260619 00:57:35.230397 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:35.213036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.2:39387)
0619 00:57:35.213624 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:35.213638 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:35.213676 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:57:35.215242 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:35.215255 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:35.215707 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:35.216108 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:35.217675 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.218911 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.220252 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.221056 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.221269 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:35.223903 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:35.223912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:35.223929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:35.223969 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:35.226904 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.227621 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.227628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.227636 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.227816 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.228193 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.228199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.228202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.228740 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:35.229119 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:35.229670 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 00:57:35.230102 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":258,"threads_started":1}
I20260619 00:57:35.231894 17327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:35.318230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17204
I20260619 00:57:35.335788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17335
I20260619 00:57:35.354523 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:35.336906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.0.97.3:36777)
0619 00:57:35.337541 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:35.337555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:35.337596 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:57:35.339212 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:35.339223 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:35.339660 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:35.340033 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:35.341679 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.342945 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.344192 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:35.344952 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:35.345149 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:35.347904 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:35.347912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:35.347929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:35.347972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:35.351053 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.351773 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.351783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.351798 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.351978 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:35.352366 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:35.352372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:35.352375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:35.353002 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:35.353352 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:35.353751 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 00:57:35.354213 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":283,"threads_started":1}
I20260619 00:57:35.355993 17458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:35.440217 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17335
2026-06-19T00:57:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64477 ms)
[----------] 30 tests from SecurityITest (303984 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:57:35 dist-test-slave-r7bh krb5kdc[17521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:57:35 dist-test-slave-r7bh krb5kdc[17521](info): set up 2 sockets
Jun 19 00:57:35 dist-test-slave-r7bh krb5kdc[17521](info): commencing operation
krb5kdc: starting...
W20260619 00:57:36.545039 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:57:36 dist-test-slave-r7bh krb5kdc[17521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:57:36.798405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43933
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:37.137260 17534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:37.137625 17534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:37.137804 17534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:37.145351 17534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:37.145524 17534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:37.145640 17534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:37.145740 17534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:37.145829 17534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:37.156076 17534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:37.157495 17534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:37.158834 17534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:37.171437 17540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:37.171511 17539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:37.171458 17542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:37.171844 17534 server_base.cc:1034] running on GCE node
I20260619 00:57:37.173135 17534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:37.174644 17534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:37.175869 17534 hybrid_clock.cc:648] HybridClock initialized: now 1781830657175791 us; error 55 us; skew 500 ppm
I20260619 00:57:37.176265 17534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:37 dist-test-slave-r7bh krb5kdc[17521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:37.184620 17534 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:57:37.186216 17534 webserver.cc:460] Webserver started at http://127.0.97.62:46565/ using document root <none> and password file <none>
I20260619 00:57:37.186944 17534 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:37.187074 17534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:37.187399 17534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:37.190469 17534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "8a750db596214c0cbc3da05a5b1f5e1e"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "61426cdda6314983c19e88505de607a2"
server_key_iv: "15a4a59cc9d0768f8b21297f190a1e8b"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.191231 17534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a750db596214c0cbc3da05a5b1f5e1e"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "61426cdda6314983c19e88505de607a2"
server_key_iv: "15a4a59cc9d0768f8b21297f190a1e8b"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.196303 17534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260619 00:57:37.199882 17549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:37.201388 17534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:37.201572 17534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "8a750db596214c0cbc3da05a5b1f5e1e"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "61426cdda6314983c19e88505de607a2"
server_key_iv: "15a4a59cc9d0768f8b21297f190a1e8b"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.201774 17534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:37.217438 17534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:37.222059 17534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:37.222386 17534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:37.242043 17534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43933
I20260619 00:57:37.242077 17600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43933 every 8 connection(s)
I20260619 00:57:37.243489 17534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:37.247277 17601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:37.253857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17534
I20260619 00:57:37.254076 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:37.254745 388 external_mini_cluster.cc:1428] Setting key 4b6846f78c1b63a9ebb4a27a77cc2d88
I20260619 00:57:37.258823 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: Bootstrap starting.
I20260619 00:57:37.262980 17601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:37.264413 17601 log.cc:824] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:37.267863 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: No bootstrap required, opened a new log
Jun 19 00:57:37 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:37.278198 17601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } }
I20260619 00:57:37.278717 17601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:37.278834 17601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a750db596214c0cbc3da05a5b1f5e1e, State: Initialized, Role: FOLLOWER
I20260619 00:57:37.279553 17601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } }
I20260619 00:57:37.279881 17601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:37.280102 17601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:37.280287 17601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:37.282023 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:37.256852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.97.62:43933)
0619 00:57:37.257639 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:37.257653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:37.257707 (+ 54us) server_negotiation.cc:407] Connection header received
0619 00:57:37.260234 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:37.260266 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:37.260913 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:37.261435 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:37.262899 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.265006 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.266725 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.267774 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.267992 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:37.272711 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:37.272725 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:37.272752 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:37.272824 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:37.277042 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:37.277785 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:37.277794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:37.277803 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:37.278025 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:37.278413 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:37.278418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:37.278421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:37.279217 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:37.279523 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:37.279979 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 00:57:37.280480 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":329,"threads_started":1}
I20260619 00:57:37.283702 17601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } }
I20260619 00:57:37.284257 17601 leader_election.cc:302] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a750db596214c0cbc3da05a5b1f5e1e; no voters:
I20260619 00:57:37.284926 17601 leader_election.cc:288] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:37.285054 17606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:37.285764 17606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [term 1 LEADER]: Becoming Leader. State: Replica: 8a750db596214c0cbc3da05a5b1f5e1e, State: Running, Role: LEADER
I20260619 00:57:37.286541 17606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } }
I20260619 00:57:37.287521 17601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:37.289330 17607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a750db596214c0cbc3da05a5b1f5e1e. Latest consensus state: current_term: 1 leader_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } } }
I20260619 00:57:37.289811 17607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:37.290548 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a750db596214c0cbc3da05a5b1f5e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43933 } } }
I20260619 00:57:37.291041 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:37.292165 17611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:37.297914 17611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:37.308470 17611 catalog_manager.cc:1347] Generated new cluster ID: b9cde14941b54f4e9f06dece191762d2
I20260619 00:57:37.308660 17611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:37.322145 17611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:37.323551 17611 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:37.329396 17611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: Generated new TSK 0
I20260619 00:57:37.330243 17611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:57:37.392971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:37.737062 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:37.737391 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:37.737569 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:37.744853 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:37.745034 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:37.745237 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:37.755348 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:37.756798 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:37.758297 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:37.771901 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:37.771967 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:37.771991 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:37.773149 17627 server_base.cc:1034] running on GCE node
I20260619 00:57:37.773859 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:37.774847 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:37.776077 17627 hybrid_clock.cc:648] HybridClock initialized: now 1781830657776020 us; error 60 us; skew 500 ppm
I20260619 00:57:37.776413 17627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:37 dist-test-slave-r7bh krb5kdc[17521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:37.779654 17627 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:57:37.781522 17627 webserver.cc:460] Webserver started at http://127.0.97.1:42985/ using document root <none> and password file <none>
I20260619 00:57:37.782208 17627 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:37.782311 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:37.782588 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:37.785753 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "49a03ab58b7e43bca149f07a5a62b4d0"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "02e395488c281b4f926ba4899f2dbc8e"
server_key_iv: "be56499ce2f929424aa86cb516001bca"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.786497 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49a03ab58b7e43bca149f07a5a62b4d0"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "02e395488c281b4f926ba4899f2dbc8e"
server_key_iv: "be56499ce2f929424aa86cb516001bca"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.792208 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:57:37.796089 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:37.797667 17627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:37.797813 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "49a03ab58b7e43bca149f07a5a62b4d0"
format_stamp: "Formatted at 2026-06-19 00:57:37 on dist-test-slave-r7bh"
server_key: "02e395488c281b4f926ba4899f2dbc8e"
server_key_iv: "be56499ce2f929424aa86cb516001bca"
server_key_version: "encryptionkey@0"
I20260619 00:57:37.797991 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:37.813091 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:37.817266 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:37.817520 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:37.818464 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:37.819787 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:37.819861 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:37.819962 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:37.820011 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:37.854406 17627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33605
I20260619 00:57:37.854517 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33605 every 8 connection(s)
I20260619 00:57:37.855767 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:37.861054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17627
I20260619 00:57:37.861311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:37.862206 388 external_mini_cluster.cc:1428] Setting key 28c9bf62a6023165b8418ea3b50796a4
Jun 19 00:57:37 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:37.881490 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:37.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57201 (local address 127.0.97.62:43933)
0619 00:57:37.859061 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:37.859066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:37.859938 (+ 872us) server_negotiation.cc:407] Connection header received
0619 00:57:37.862809 (+ 2871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:37.862814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:37.862891 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:37.863036 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:37.865136 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.865844 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.868571 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.869070 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.869173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:37.874068 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:37.874079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:37.874086 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:37.874133 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:37.877874 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:37.878656 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:37.878665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:37.878671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:37.878819 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:37.879310 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:37.879318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:37.879324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:37.879713 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:37.879897 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:37.880683 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 00:57:37.880991 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
I20260619 00:57:37.882239 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:37.858935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43933 (local address 127.0.97.1:57201)
0619 00:57:37.859724 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:37.859774 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:37.862405 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:37.863339 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:37.863355 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:37.863996 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:37.864934 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:37.864965 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.866051 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.866060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:37.868397 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:37.868418 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:37.869325 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:37.869336 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:37.869714 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:37.870539 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:37.870579 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:37.873796 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:37.878042 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:37.878050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:37.878064 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:37.878506 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:37.878969 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:37.878974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:37.878976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:37.879181 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:37.879868 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:37.879887 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:37.880269 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 00:57:37.880524 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 00:57:37.881037 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260619 00:57:37.883994 17755 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43933
I20260619 00:57:37.884332 17755 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:37.885159 17755 heartbeater.cc:511] Master 127.0.97.62:43933 requested a full tablet report, sending...
I20260619 00:57:37.887493 17566 ts_manager.cc:194] Registered new tserver with Master: 49a03ab58b7e43bca149f07a5a62b4d0 (127.0.97.1:33605)
I20260619 00:57:37.889240 17566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57201
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:57:37.920125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:38.265798 17761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:38.266211 17761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:38.266398 17761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:38.274082 17761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:38.274260 17761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:38.274451 17761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:38.284539 17761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:38.285979 17761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:38.287461 17761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:38.301293 17769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:38.301262 17767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:38.301373 17766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:38.306960 17761 server_base.cc:1034] running on GCE node
I20260619 00:57:38.307672 17761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:38.308679 17761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:38.309918 17761 hybrid_clock.cc:648] HybridClock initialized: now 1781830658309857 us; error 47 us; skew 500 ppm
I20260619 00:57:38.310243 17761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:38 dist-test-slave-r7bh krb5kdc[17521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:38.314393 17761 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:57:38.316046 17761 webserver.cc:460] Webserver started at http://127.0.97.2:38185/ using document root <none> and password file <none>
I20260619 00:57:38.316752 17761 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:38.316915 17761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:38.317215 17761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:38.320222 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "a607cd17ac2941a0b95df651fa2ab8ae"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "33824b7d0bcc897f8a1db15e99daa805"
server_key_iv: "417a826ade622a2b5237b8b67a209907"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.321043 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a607cd17ac2941a0b95df651fa2ab8ae"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "33824b7d0bcc897f8a1db15e99daa805"
server_key_iv: "417a826ade622a2b5237b8b67a209907"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.326402 17761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 00:57:38.330061 17776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.331508 17761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:38.331650 17761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "a607cd17ac2941a0b95df651fa2ab8ae"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "33824b7d0bcc897f8a1db15e99daa805"
server_key_iv: "417a826ade622a2b5237b8b67a209907"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.331836 17761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:38.351050 17761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:38.355819 17761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:38.356067 17761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:38.357071 17761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:38.358392 17761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:38.358465 17761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.358551 17761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:38.358596 17761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.392580 17761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33733
I20260619 00:57:38.392624 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33733 every 8 connection(s)
I20260619 00:57:38.394006 17761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:38.395293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17761
I20260619 00:57:38.395481 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:38.395920 388 external_mini_cluster.cc:1428] Setting key 19a8615721e6a355a0379b74b3f0822f
Jun 19 00:57:38 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:38.418490 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:38.396873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38811 (local address 127.0.97.62:43933)
0619 00:57:38.397307 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:38.397312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:38.398793 (+ 1481us) server_negotiation.cc:407] Connection header received
0619 00:57:38.400727 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:38.400736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:38.400860 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:38.400987 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:38.402688 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.403946 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.405740 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.406486 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.406593 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:38.411071 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:38.411081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:38.411089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:38.411139 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:38.414838 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.415695 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.415704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.415710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.415876 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.416321 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.416329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.416335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.416737 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:38.416924 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:38.417694 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 00:57:38.417964 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":208,"threads_started":1}
I20260619 00:57:38.419162 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:38.397556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43933 (local address 127.0.97.2:38811)
0619 00:57:38.398606 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:38.398651 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:38.400428 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:38.401181 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:38.401190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:38.401837 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:38.402499 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:38.402519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.404124 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.404128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:38.405587 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:38.405600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.406741 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.406749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:38.407086 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:38.407915 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:38.407955 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:38.410782 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:38.415022 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:38.415030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:38.415046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:38.415537 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:38.415997 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:38.416000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:38.416002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:38.416187 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:38.416928 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:38.416934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:38.417273 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 00:57:38.417533 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 00:57:38.417951 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":576,"threads_started":1}
I20260619 00:57:38.420933 17889 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43933
I20260619 00:57:38.421263 17889 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:38.422067 17889 heartbeater.cc:511] Master 127.0.97.62:43933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:57:38.423738 17566 ts_manager.cc:194] Registered new tserver with Master: a607cd17ac2941a0b95df651fa2ab8ae (127.0.97.2:33733)
I20260619 00:57:38.424610 17566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:57:38.455931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:38.800271 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:38.800632 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:38.800863 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:38.808650 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:38.808851 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:38.809046 17895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:38.819350 17895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:38.820768 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:38.822232 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:38.836674 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:38.836728 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:38.836705 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:38.837579 17895 server_base.cc:1034] running on GCE node
I20260619 00:57:38.838274 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:38.839334 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:38.840586 17895 hybrid_clock.cc:648] HybridClock initialized: now 1781830658840538 us; error 41 us; skew 500 ppm
I20260619 00:57:38.841022 17895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:38 dist-test-slave-r7bh krb5kdc[17521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:38.844486 17895 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:57:38.846158 17895 webserver.cc:460] Webserver started at http://127.0.97.3:44667/ using document root <none> and password file <none>
I20260619 00:57:38.846906 17895 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:38.847051 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:38.847379 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:38.850587 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "cfc12afedbe441c2b94381415f48f800"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "ce4a5c15a8b78eb5643d7ac6434d46cf"
server_key_iv: "923c547fde540acce03cf948cac20a17"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.851393 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cfc12afedbe441c2b94381415f48f800"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "ce4a5c15a8b78eb5643d7ac6434d46cf"
server_key_iv: "923c547fde540acce03cf948cac20a17"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.857134 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 00:57:38.860980 17910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.862491 17895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:38.862699 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "cfc12afedbe441c2b94381415f48f800"
format_stamp: "Formatted at 2026-06-19 00:57:38 on dist-test-slave-r7bh"
server_key: "ce4a5c15a8b78eb5643d7ac6434d46cf"
server_key_iv: "923c547fde540acce03cf948cac20a17"
server_key_version: "encryptionkey@0"
I20260619 00:57:38.862944 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:38.887717 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:38.892339 17755 heartbeater.cc:503] Master 127.0.97.62:43933 was elected leader, sending a full tablet report...
I20260619 00:57:38.892743 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:38.893131 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:38.894361 17895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:38.895880 17895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:38.895987 17895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.896138 17895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:38.896214 17895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:38.930878 17895 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40979
I20260619 00:57:38.930994 18022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40979 every 8 connection(s)
I20260619 00:57:38.932278 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:38.941556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 17895
I20260619 00:57:38.941752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:38.942201 388 external_mini_cluster.cc:1428] Setting key e460763f829da49f4e1750ec69676ce5
Jun 19 00:57:38 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:38.956647 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:38.934987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53855 (local address 127.0.97.62:43933)
0619 00:57:38.935497 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:38.935503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:38.936578 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 00:57:38.938494 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:38.938499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:38.938581 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:38.938686 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:38.940316 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.941207 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.943324 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.943920 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.944021 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:38.949036 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:38.949044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:38.949050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:38.949095 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:38.952882 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.954007 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.954017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.954022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.954190 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.954686 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.954691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.954695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.955003 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:38.955183 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:38.955959 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 00:57:38.956214 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":214,"threads_started":1}
I20260619 00:57:38.957722 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:38.935630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43933 (local address 127.0.97.3:53855)
0619 00:57:38.936393 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:38.936433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:38.938236 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:38.938975 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:38.938983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:38.939553 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:38.940144 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:38.940161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.941409 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.941414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:38.943150 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:38.943167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.944187 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.944197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:38.944597 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:38.945503 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:38.945539 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:38.948709 (+ 3170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:38.953074 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:38.953085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:38.953106 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:38.953824 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:38.954319 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:38.954325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:38.954329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:38.954553 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:38.955159 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:38.955169 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:38.955556 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 00:57:38.955854 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 00:57:38.956461 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":306,"threads_started":1}
I20260619 00:57:38.959290 18023 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43933
I20260619 00:57:38.959659 18023 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:38.960424 18023 heartbeater.cc:511] Master 127.0.97.62:43933 requested a full tablet report, sending...
I20260619 00:57:38.962045 17566 ts_manager.cc:194] Registered new tserver with Master: cfc12afedbe441c2b94381415f48f800 (127.0.97.3:40979)
I20260619 00:57:38.962903 17566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53855
I20260619 00:57:38.965143 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:38.983559 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:38.970288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.0.97.62:43933)
0619 00:57:38.970526 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:38.970531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:38.970978 (+ 447us) server_negotiation.cc:407] Connection header received
0619 00:57:38.971116 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:38.971120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:38.971216 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:38.971315 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:38.972866 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.973739 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.975095 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:38.975504 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:38.975604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:38.978246 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:38.978252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:38.978255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:38.978291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:38.980899 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.981672 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.981677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.981679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.981816 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:38.982175 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:38.982178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:38.982180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:38.982526 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:38.982716 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:38.983094 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 00:57:38.983317 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:57:38.988029 17566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:38.995462 17566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:39.023329 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.009216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.0.97.1:33605)
0619 00:57:39.012929 (+ 3713us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.012942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.013525 (+ 583us) server_negotiation.cc:407] Connection header received
0619 00:57:39.013849 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.013863 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.014170 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.015301 (+ 1131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:39.015581 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.017209 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.020441 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.021822 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.021950 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.022223 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.022783 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.022966 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3494,"thread_start_us":3348,"threads_started":1}
I20260619 00:57:39.023408 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.008739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33605 (local address 127.0.0.1:36396)
0619 00:57:39.013404 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:39.013423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:39.013630 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:39.014449 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:39.014457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:39.014488 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:39.014994 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.015011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.017361 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.017366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.020225 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.020240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.022374 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.022383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.022542 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.022591 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 00:57:39.022733 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 00:57:39.022966 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4508,"mutex_wait_us":254,"thread_start_us":445,"threads_started":1}
I20260619 00:57:39.024113 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.008918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.97.2:33733)
0619 00:57:39.013230 (+ 4312us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.013241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.013273 (+ 32us) server_negotiation.cc:407] Connection header received
0619 00:57:39.013410 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.013421 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.014590 (+ 1169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.015616 (+ 1026us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:39.015797 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.017114 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.020425 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.022039 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.023167 (+ 1128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.023356 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.023438 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.023683 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4155,"thread_start_us":4047,"threads_started":1}
I20260619 00:57:39.025441 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.008682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33733 (local address 127.0.0.1:47722)
0619 00:57:39.009661 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:39.009723 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:39.009952 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:39.014840 (+ 4888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:39.014848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:39.014886 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:39.015423 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.015435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.017238 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.017246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.020232 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.020240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.022290 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.022303 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.022561 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.022616 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 00:57:39.022699 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 00:57:39.022959 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":116,"thread_start_us":4400,"threads_started":1}
I20260619 00:57:39.025499 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.013891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.0.97.3:40979)
0619 00:57:39.015681 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.015691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.015728 (+ 37us) server_negotiation.cc:407] Connection header received
0619 00:57:39.015834 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.015843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.016081 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.016290 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:39.016904 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.018844 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.021396 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.022926 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.023076 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.023346 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.024934 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.025132 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":1422,"threads_started":1}
I20260619 00:57:39.025517 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.013567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40979 (local address 127.0.0.1:59628)
0619 00:57:39.014151 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:39.014170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:39.014315 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:39.016304 (+ 1989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:39.016309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:39.016333 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:39.016672 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.016684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.018995 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.019002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.020887 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.020904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.023519 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.023526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.024697 (+ 1171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.024739 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 00:57:39.024879 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 00:57:39.025030 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":2,"thread_start_us":298,"threads_started":1}
I20260619 00:57:39.027915 17690 tablet_service.cc:1467] Processing CreateTablet for tablet a3190f20043e4bc38f2567d411ea1044 (DEFAULT_TABLE table=test-table [id=4243979c3e574e558f375e299e1da8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:39.029552 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3190f20043e4bc38f2567d411ea1044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:39.029844 17824 tablet_service.cc:1467] Processing CreateTablet for tablet a3190f20043e4bc38f2567d411ea1044 (DEFAULT_TABLE table=test-table [id=4243979c3e574e558f375e299e1da8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:39.030046 17958 tablet_service.cc:1467] Processing CreateTablet for tablet a3190f20043e4bc38f2567d411ea1044 (DEFAULT_TABLE table=test-table [id=4243979c3e574e558f375e299e1da8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:39.031224 17824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3190f20043e4bc38f2567d411ea1044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:39.031405 17958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3190f20043e4bc38f2567d411ea1044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:39.043696 18042 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Bootstrap starting.
I20260619 00:57:39.045930 18044 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Bootstrap starting.
I20260619 00:57:39.049216 18042 tablet_bootstrap.cc:654] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:39.049563 18043 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Bootstrap starting.
I20260619 00:57:39.049876 18044 tablet_bootstrap.cc:654] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:39.050917 18042 log.cc:824] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:39.051118 18044 log.cc:824] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:39.052892 18043 tablet_bootstrap.cc:654] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:39.054162 18042 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: No bootstrap required, opened a new log
I20260619 00:57:39.054162 18043 log.cc:824] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:39.054461 18044 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: No bootstrap required, opened a new log
I20260619 00:57:39.054467 18042 ts_tablet_manager.cc:1397] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 00:57:39.054754 18044 ts_tablet_manager.cc:1397] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 00:57:39.057983 18043 tablet_bootstrap.cc:492] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: No bootstrap required, opened a new log
I20260619 00:57:39.058271 18043 ts_tablet_manager.cc:1397] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 00:57:39.062613 18042 raft_consensus.cc:348] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.063050 18042 raft_consensus.cc:374] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:39.063166 18042 raft_consensus.cc:729] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a607cd17ac2941a0b95df651fa2ab8ae, State: Initialized, Role: FOLLOWER
I20260619 00:57:39.063853 18042 consensus_queue.cc:260] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.065126 17889 heartbeater.cc:503] Master 127.0.97.62:43933 was elected leader, sending a full tablet report...
I20260619 00:57:39.065423 18044 raft_consensus.cc:348] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.065881 18044 raft_consensus.cc:374] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:39.065927 18042 ts_tablet_manager.cc:1428] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 00:57:39.065977 18044 raft_consensus.cc:729] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc12afedbe441c2b94381415f48f800, State: Initialized, Role: FOLLOWER
I20260619 00:57:39.066808 18044 consensus_queue.cc:260] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.070343 18023 heartbeater.cc:503] Master 127.0.97.62:43933 was elected leader, sending a full tablet report...
I20260619 00:57:39.071056 18044 ts_tablet_manager.cc:1428] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260619 00:57:39.070721 18043 raft_consensus.cc:348] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.071219 18043 raft_consensus.cc:374] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:39.071334 18043 raft_consensus.cc:729] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a03ab58b7e43bca149f07a5a62b4d0, State: Initialized, Role: FOLLOWER
I20260619 00:57:39.071998 18043 consensus_queue.cc:260] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.073975 18043 ts_tablet_manager.cc:1428] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260619 00:57:39.108239 17756 tablet.cc:2369] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:39.146140 17890 tablet.cc:2369] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:39.183995 18024 tablet.cc:2369] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:39.238408 18049 raft_consensus.cc:482] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:39.238770 18049 raft_consensus.cc:504] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.240567 18049 leader_election.cc:288] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a607cd17ac2941a0b95df651fa2ab8ae (127.0.97.2:33733), 49a03ab58b7e43bca149f07a5a62b4d0 (127.0.97.1:33605)
I20260619 00:57:39.247407 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.241150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46617 (local address 127.0.97.2:33733)
0619 00:57:39.241351 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.241357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.241685 (+ 328us) server_negotiation.cc:407] Connection header received
0619 00:57:39.241777 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.241786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.241864 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.241972 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:39.242454 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.243635 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.245479 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.246397 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.246542 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.246655 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.246976 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.247132 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:57:39.247440 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.240952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33733 (local address 127.0.97.3:46617)
0619 00:57:39.241329 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:39.241356 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:39.241534 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:39.242004 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:39.242008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:39.242036 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:39.242286 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.242297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.243782 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.243787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.245288 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.245302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.246576 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.246582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.246782 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.246828 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 00:57:39.246922 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 00:57:39.247128 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":23,"spinlock_wait_cycles":67456}
I20260619 00:57:39.248646 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.241061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52251 (local address 127.0.97.1:33605)
0619 00:57:39.241286 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.241293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.241965 (+ 672us) server_negotiation.cc:407] Connection header received
0619 00:57:39.242288 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.242294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.242387 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.242472 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:39.243029 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.243773 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.246227 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.247438 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.247578 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.247703 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.248255 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.248403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:57:39.248699 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.241029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33605 (local address 127.0.97.3:52251)
0619 00:57:39.241845 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:39.241864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:39.242096 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:39.242482 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:39.242496 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:39.242519 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:39.242847 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.242856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.243933 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.243939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.246047 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:39.246062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.247865 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.247873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:39.248064 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.248094 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:57:39.248192 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 00:57:39.248343 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":462,"threads_started":1}
I20260619 00:57:39.248759 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3190f20043e4bc38f2567d411ea1044" candidate_uuid: "cfc12afedbe441c2b94381415f48f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" is_pre_election: true
I20260619 00:57:39.249483 17844 raft_consensus.cc:2454] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc12afedbe441c2b94381415f48f800 in term 0.
I20260619 00:57:39.249764 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3190f20043e4bc38f2567d411ea1044" candidate_uuid: "cfc12afedbe441c2b94381415f48f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" is_pre_election: true
I20260619 00:57:39.250291 17913 leader_election.cc:302] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a607cd17ac2941a0b95df651fa2ab8ae, cfc12afedbe441c2b94381415f48f800; no voters:
I20260619 00:57:39.250370 17710 raft_consensus.cc:2454] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc12afedbe441c2b94381415f48f800 in term 0.
I20260619 00:57:39.250763 18049 raft_consensus.cc:2781] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:39.250931 18049 raft_consensus.cc:482] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:39.251017 18049 raft_consensus.cc:3037] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:39.254292 18049 raft_consensus.cc:504] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.255170 18049 leader_election.cc:288] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [CANDIDATE]: Term 1 election: Requested vote from peers a607cd17ac2941a0b95df651fa2ab8ae (127.0.97.2:33733), 49a03ab58b7e43bca149f07a5a62b4d0 (127.0.97.1:33605)
I20260619 00:57:39.255851 17844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3190f20043e4bc38f2567d411ea1044" candidate_uuid: "cfc12afedbe441c2b94381415f48f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a607cd17ac2941a0b95df651fa2ab8ae"
I20260619 00:57:39.256112 17844 raft_consensus.cc:3037] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:39.256121 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3190f20043e4bc38f2567d411ea1044" candidate_uuid: "cfc12afedbe441c2b94381415f48f800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a03ab58b7e43bca149f07a5a62b4d0"
I20260619 00:57:39.256328 17710 raft_consensus.cc:3037] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:39.259560 17844 raft_consensus.cc:2454] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc12afedbe441c2b94381415f48f800 in term 1.
I20260619 00:57:39.259560 17710 raft_consensus.cc:2454] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc12afedbe441c2b94381415f48f800 in term 1.
I20260619 00:57:39.260049 17913 leader_election.cc:302] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a607cd17ac2941a0b95df651fa2ab8ae, cfc12afedbe441c2b94381415f48f800; no voters:
I20260619 00:57:39.260383 18049 raft_consensus.cc:2781] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:39.260921 18049 raft_consensus.cc:686] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 1 LEADER]: Becoming Leader. State: Replica: cfc12afedbe441c2b94381415f48f800, State: Running, Role: LEADER
I20260619 00:57:39.261430 18049 consensus_queue.cc:237] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260619 00:57:39.266654 17566 catalog_manager.cc:5495] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc12afedbe441c2b94381415f48f800 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cfc12afedbe441c2b94381415f48f800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc12afedbe441c2b94381415f48f800" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } health_report { overall_health: HEALTHY } } }
I20260619 00:57:39.324267 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.319058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.97.3:40979)
0619 00:57:39.319245 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.319249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.319287 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:57:39.319390 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.319394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.319474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.319601 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:39.319962 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.320685 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.322187 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.322599 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.322713 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.323215 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:39.323734 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:39.323866 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.323991 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.324061 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:57:39.333112 17844 raft_consensus.cc:1261] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 1 FOLLOWER]: Refusing update from remote peer cfc12afedbe441c2b94381415f48f800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:39.333101 17710 raft_consensus.cc:1261] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 1 FOLLOWER]: Refusing update from remote peer cfc12afedbe441c2b94381415f48f800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:39.334401 18049 consensus_queue.cc:1035] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a607cd17ac2941a0b95df651fa2ab8ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:39.334779 18052 consensus_queue.cc:1035] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a03ab58b7e43bca149f07a5a62b4d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:39.345223 18055 mvcc.cc:205] Tried to move back new op lower bound from 7298378380616613888 to 7298378380338868224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378380616613888}]
I20260619 00:57:39.349479 18056 mvcc.cc:205] Tried to move back new op lower bound from 7298378380616613888 to 7298378380338868224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378380616613888}]
I20260619 00:57:39.358263 18057 mvcc.cc:205] Tried to move back new op lower bound from 7298378380616613888 to 7298378380338868224. Current Snapshot: MvccSnapshot[applied={T|T < 7298378380616613888}]
I20260619 00:57:39.365521 17565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:39.365890 17565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:39.370658 17565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a750db596214c0cbc3da05a5b1f5e1e: Sending DeleteTablet for 3 replicas of tablet a3190f20043e4bc38f2567d411ea1044
I20260619 00:57:39.372205 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet a3190f20043e4bc38f2567d411ea1044 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36396
I20260619 00:57:39.372314 17824 tablet_service.cc:1514] Processing DeleteTablet for tablet a3190f20043e4bc38f2567d411ea1044 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47722
I20260619 00:57:39.372390 17958 tablet_service.cc:1514] Processing DeleteTablet for tablet a3190f20043e4bc38f2567d411ea1044 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59628
I20260619 00:57:39.375669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17627
I20260619 00:57:39.388530 18067 tablet_replica.cc:330] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: stopping tablet replica
I20260619 00:57:39.389856 18067 raft_consensus.cc:2229] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:39.390911 18067 raft_consensus.cc:2258] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:39.407048 18065 tablet_replica.cc:330] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: stopping tablet replica
I20260619 00:57:39.407598 18065 raft_consensus.cc:2229] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:39.408224 18065 raft_consensus.cc:2258] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:39.411567 18067 ts_tablet_manager.cc:1905] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:39.417480 18065 ts_tablet_manager.cc:1905] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:39.418272 18066 tablet_replica.cc:330] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: stopping tablet replica
I20260619 00:57:39.418798 18066 raft_consensus.cc:2229] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:39.419445 18066 raft_consensus.cc:2258] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 00:57:39 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:57:39.426698 18066 ts_tablet_manager.cc:1905] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:39.426846 18065 ts_tablet_manager.cc:1918] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:39.427042 18065 log.cc:1196] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/a3190f20043e4bc38f2567d411ea1044
I20260619 00:57:39.427778 18065 ts_tablet_manager.cc:1939] T a3190f20043e4bc38f2567d411ea1044 P a607cd17ac2941a0b95df651fa2ab8ae: Deleting consensus metadata
I20260619 00:57:39.428432 18067 ts_tablet_manager.cc:1918] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:39.428637 18067 log.cc:1196] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/a3190f20043e4bc38f2567d411ea1044
I20260619 00:57:39.429265 18067 ts_tablet_manager.cc:1939] T a3190f20043e4bc38f2567d411ea1044 P cfc12afedbe441c2b94381415f48f800: Deleting consensus metadata
I20260619 00:57:39.431474 17552 catalog_manager.cc:4841] TS cfc12afedbe441c2b94381415f48f800 (127.0.97.3:40979): tablet a3190f20043e4bc38f2567d411ea1044 (table test-table [id=4243979c3e574e558f375e299e1da8ac]) successfully deleted
I20260619 00:57:39.433531 17552 catalog_manager.cc:4841] TS a607cd17ac2941a0b95df651fa2ab8ae (127.0.97.2:33733): tablet a3190f20043e4bc38f2567d411ea1044 (table test-table [id=4243979c3e574e558f375e299e1da8ac]) successfully deleted
I20260619 00:57:39.441792 18066 ts_tablet_manager.cc:1918] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:39.442039 18066 log.cc:1196] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/a3190f20043e4bc38f2567d411ea1044
I20260619 00:57:39.443516 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.377854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.97.1:33605)
0619 00:57:39.378167 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.378179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.403968 (+ 25789us) server_negotiation.cc:407] Connection header received
0619 00:57:39.404559 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.404567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.404670 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.404766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:39.411159 (+ 6393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.412534 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.415395 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.416226 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.416363 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.433520 (+ 17157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:39.433544 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:39.433567 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:39.433625 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:39.437835 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.439059 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.439073 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.439077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.439323 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.440068 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.440075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.440080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.440719 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:39.441119 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.442396 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.442739 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 00:57:39.442812 18066 ts_tablet_manager.cc:1939] T a3190f20043e4bc38f2567d411ea1044 P 49a03ab58b7e43bca149f07a5a62b4d0: Deleting consensus metadata
I20260619 00:57:39.444450 17750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:39.588999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17627
I20260619 00:57:39.589037 17551 catalog_manager.cc:4841] TS 49a03ab58b7e43bca149f07a5a62b4d0 (127.0.97.1:33605): tablet a3190f20043e4bc38f2567d411ea1044 (table test-table [id=4243979c3e574e558f375e299e1da8ac]) successfully deleted
I20260619 00:57:39.607509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17761
Jun 19 00:57:39 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:57:39.625396 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.608385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.0.97.2:33733)
0619 00:57:39.608558 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.608563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.608619 (+ 56us) server_negotiation.cc:407] Connection header received
0619 00:57:39.608877 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.608881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.608947 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.609037 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:39.610709 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.611383 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.613448 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.613871 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.613979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.619159 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:39.619176 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:39.619194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:39.619232 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:39.622248 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.623027 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.623034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.623036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.623223 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.623732 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.623738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.623742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.624149 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:39.624457 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.624755 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.625013 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:57:39.626168 17884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:39.716709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17761
I20260619 00:57:39.740607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17895
Jun 19 00:57:39 dist-test-slave-r7bh krb5kdc[17521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:57:39.758160 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:39.741528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.0.97.3:40979)
0619 00:57:39.741715 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:39.741722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:39.741822 (+ 100us) server_negotiation.cc:407] Connection header received
0619 00:57:39.742038 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:39.742043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:39.742127 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:39.742246 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:39.743549 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.744399 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.745920 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:39.746378 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:39.746531 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:39.751139 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:39.751155 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:39.751172 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:39.751229 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:39.754800 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.755651 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.755664 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.755674 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.755904 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:39.756342 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:39.756348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:39.756351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:39.756736 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:39.757180 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:39.757462 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 00:57:39.757708 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 00:57:39.758994 18018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:39.888296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17895
I20260619 00:57:39.908805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 17534
I20260619 00:57:39.910028 17596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:40.022359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 17534
2026-06-19T00:57:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4589 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T00:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:40Z Disabled control of system clock
I20260619 00:57:40.104283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34797
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:40.441761 18076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:40.442062 18076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:40.442215 18076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:40.449527 18076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:40.449636 18076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:40.449708 18076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:40.449803 18076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:40.449890 18076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:40.459246 18076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:40.460544 18076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:40.462389 18076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:40.474648 18082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:40.474763 18081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:40.474738 18084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:40.475546 18076 server_base.cc:1034] running on GCE node
I20260619 00:57:40.476392 18076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:40.478065 18076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:40.479270 18076 hybrid_clock.cc:648] HybridClock initialized: now 1781830660479233 us; error 32 us; skew 500 ppm
I20260619 00:57:40.479656 18076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:40.482008 18076 webserver.cc:460] Webserver started at http://127.0.97.62:44613/ using document root <none> and password file <none>
I20260619 00:57:40.482692 18076 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:40.482779 18076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:40.483088 18076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:40.486020 18076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "8f9ca98943ff4ff4bfa55e513cce8844"
format_stamp: "Formatted at 2026-06-19 00:57:40 on dist-test-slave-r7bh"
server_key: "c162eff75ab59f3a0a3957fd60ce902d"
server_key_iv: "ad97d885c7dae50a4a36a7d63823e17f"
server_key_version: "encryptionkey@0"
I20260619 00:57:40.486881 18076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f9ca98943ff4ff4bfa55e513cce8844"
format_stamp: "Formatted at 2026-06-19 00:57:40 on dist-test-slave-r7bh"
server_key: "c162eff75ab59f3a0a3957fd60ce902d"
server_key_iv: "ad97d885c7dae50a4a36a7d63823e17f"
server_key_version: "encryptionkey@0"
I20260619 00:57:40.492209 18076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:57:40.495697 18090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:40.497210 18076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:40.497432 18076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "8f9ca98943ff4ff4bfa55e513cce8844"
format_stamp: "Formatted at 2026-06-19 00:57:40 on dist-test-slave-r7bh"
server_key: "c162eff75ab59f3a0a3957fd60ce902d"
server_key_iv: "ad97d885c7dae50a4a36a7d63823e17f"
server_key_version: "encryptionkey@0"
I20260619 00:57:40.497654 18076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:40.525214 18076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:40.526163 18076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:40.526476 18076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:40.545439 18076 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34797
I20260619 00:57:40.545459 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34797 every 8 connection(s)
I20260619 00:57:40.546896 18076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:40.549183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18076
I20260619 00:57:40.549362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:40.549826 388 external_mini_cluster.cc:1428] Setting key eb48c5dd709fb51020137dd74ae4ba07
I20260619 00:57:40.551244 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:40.565089 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:40.551766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.0.97.62:34797)
0619 00:57:40.552783 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:40.552793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:40.552893 (+ 100us) server_negotiation.cc:407] Connection header received
0619 00:57:40.554544 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:40.554567 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:40.555062 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:40.555463 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:40.556330 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:40.557845 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:40.560149 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:40.560928 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:40.561070 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:40.562525 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:40.562533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:40.562551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:40.563137 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:40.563214 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:40.563236 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:40.563329 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:40.563598 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 00:57:40.563989 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":375,"threads_started":1}
I20260619 00:57:40.566928 18142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: Bootstrap starting.
I20260619 00:57:40.570600 18142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:40.571645 18142 log.cc:824] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:40.574460 18142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: No bootstrap required, opened a new log
I20260619 00:57:40.582623 18142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } }
I20260619 00:57:40.583032 18142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:40.583148 18142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9ca98943ff4ff4bfa55e513cce8844, State: Initialized, Role: FOLLOWER
I20260619 00:57:40.583844 18142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } }
I20260619 00:57:40.584115 18142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:40.584280 18142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:40.584473 18142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:40.586867 18142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } }
I20260619 00:57:40.587427 18142 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9ca98943ff4ff4bfa55e513cce8844; no voters:
I20260619 00:57:40.588052 18142 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:40.588208 18147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:40.588996 18147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9ca98943ff4ff4bfa55e513cce8844, State: Running, Role: LEADER
I20260619 00:57:40.589550 18147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } }
I20260619 00:57:40.590559 18142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:40.592983 18149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9ca98943ff4ff4bfa55e513cce8844. Latest consensus state: current_term: 1 leader_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } } }
I20260619 00:57:40.593534 18149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:40.593003 18148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ca98943ff4ff4bfa55e513cce8844" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34797 } } }
I20260619 00:57:40.593691 18148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:40.594996 18155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:40.600572 18155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:40.610633 18155 catalog_manager.cc:1347] Generated new cluster ID: 1a9cb534dfa84790bca27a1cd020080e
I20260619 00:57:40.610795 18155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:40.645455 18155 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:40.646857 18155 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:40.656702 18155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: Generated new TSK 0
I20260619 00:57:40.657832 18155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:57:40.677219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--builtin_ntp_servers=127.0.97.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:41.048219 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:41.048681 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:41.048879 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:41.060774 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:41.060952 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:41.061157 18166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:41.079406 18166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:41.081408 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:41.083500 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:41.096908 18172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:41.096962 18174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:41.096928 18171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:41.097756 18166 server_base.cc:1034] running on GCE node
I20260619 00:57:41.098320 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:41.099326 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:41.100555 18166 hybrid_clock.cc:648] HybridClock initialized: now 1781830661100509 us; error 43 us; skew 500 ppm
I20260619 00:57:41.101052 18166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:41.103609 18166 webserver.cc:460] Webserver started at http://127.0.97.1:42813/ using document root <none> and password file <none>
I20260619 00:57:41.104355 18166 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:41.104449 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:41.104892 18166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:41.107923 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ab2e97ab7914b5bad4e94f4979b38e1"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "505f44b2955a1480e17d27bd78c12802"
server_key_iv: "e1cf2901350ce4bc22395aeda7b67c73"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.108777 18166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ab2e97ab7914b5bad4e94f4979b38e1"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "505f44b2955a1480e17d27bd78c12802"
server_key_iv: "e1cf2901350ce4bc22395aeda7b67c73"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.114410 18166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:57:41.118193 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.119683 18166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:41.119930 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "3ab2e97ab7914b5bad4e94f4979b38e1"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "505f44b2955a1480e17d27bd78c12802"
server_key_iv: "e1cf2901350ce4bc22395aeda7b67c73"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.120186 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:41.146821 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:41.147706 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:41.148010 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:41.149127 18166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:41.150673 18166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:41.150802 18166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.150949 18166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:41.151041 18166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.184552 18166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41189
I20260619 00:57:41.184644 18292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41189 every 8 connection(s)
I20260619 00:57:41.185938 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:41.186659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18166
I20260619 00:57:41.186856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:41.187480 388 external_mini_cluster.cc:1428] Setting key 7a756e98bf703eaacb570d9752eb0228
I20260619 00:57:41.190644 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--builtin_ntp_servers=127.0.97.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:41.201924 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:41.188760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46735 (local address 127.0.97.62:34797)
0619 00:57:41.189275 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:41.189281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:41.192884 (+ 3603us) server_negotiation.cc:407] Connection header received
0619 00:57:41.193724 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:41.193728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:41.193797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:41.193883 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:41.195340 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.196193 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.198711 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.199102 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.199214 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:41.200841 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:41.200850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:41.200857 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:41.200884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:41.200939 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:41.200957 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:41.201046 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:41.201347 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 00:57:41.201466 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":242,"threads_started":1}
I20260619 00:57:41.203011 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:41.189522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34797 (local address 127.0.97.1:46735)
0619 00:57:41.190650 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:41.190699 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:41.193421 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:41.194149 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:41.194163 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:41.194291 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:41.195148 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:41.195174 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.196350 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.196359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:41.198572 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:41.198590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.199363 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.199372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:41.199662 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:41.200466 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:41.200469 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:41.200509 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:41.200612 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:41.201082 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:41.201093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:41.201100 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 00:57:41.201299 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 00:57:41.201679 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":323,"threads_started":1}
I20260619 00:57:41.204660 18293 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34797
I20260619 00:57:41.205034 18293 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:41.205951 18293 heartbeater.cc:511] Master 127.0.97.62:34797 requested a full tablet report, sending...
I20260619 00:57:41.208137 18107 ts_manager.cc:194] Registered new tserver with Master: 3ab2e97ab7914b5bad4e94f4979b38e1 (127.0.97.1:41189)
I20260619 00:57:41.209676 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46735
W20260619 00:57:41.526221 18297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:41.526578 18297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:41.526751 18297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:41.534312 18297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:41.534464 18297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:41.534646 18297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:41.543857 18297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:41.545333 18297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:41.547240 18297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:41.561043 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:41.561054 18303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:41.561308 18305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:41.562727 18297 server_base.cc:1034] running on GCE node
I20260619 00:57:41.563351 18297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:41.564294 18297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:41.565569 18297 hybrid_clock.cc:648] HybridClock initialized: now 1781830661565497 us; error 48 us; skew 500 ppm
I20260619 00:57:41.565900 18297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:41.568240 18297 webserver.cc:460] Webserver started at http://127.0.97.2:37051/ using document root <none> and password file <none>
I20260619 00:57:41.569058 18297 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:41.569168 18297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:41.569492 18297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:41.572300 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "51fa655e2701442e9db7900bb2eb74d3"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "acfe7b3442feffe2c27227b0a196972e"
server_key_iv: "53f91f49601412999435bb073a50a9bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.573158 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "51fa655e2701442e9db7900bb2eb74d3"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "acfe7b3442feffe2c27227b0a196972e"
server_key_iv: "53f91f49601412999435bb073a50a9bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.578279 18297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 00:57:41.581943 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.583279 18297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:41.583461 18297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "51fa655e2701442e9db7900bb2eb74d3"
format_stamp: "Formatted at 2026-06-19 00:57:41 on dist-test-slave-r7bh"
server_key: "acfe7b3442feffe2c27227b0a196972e"
server_key_iv: "53f91f49601412999435bb073a50a9bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:41.583688 18297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:41.611676 18297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:41.612540 18297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:41.612865 18297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:41.613924 18297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:41.615444 18297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:41.615558 18297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.615693 18297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:41.615793 18297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:41.649536 18297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37785
I20260619 00:57:41.649641 18423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37785 every 8 connection(s)
I20260619 00:57:41.650897 18297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:41.659121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18297
I20260619 00:57:41.659307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:41.659808 388 external_mini_cluster.cc:1428] Setting key 86d4511e68d4d5c8e8580d9a8bbcbd04
I20260619 00:57:41.662721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--builtin_ntp_servers=127.0.97.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:41.663735 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:41.653600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37833 (local address 127.0.97.62:34797)
0619 00:57:41.653802 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:41.653807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:41.655165 (+ 1358us) server_negotiation.cc:407] Connection header received
0619 00:57:41.656897 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:41.656902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:41.657009 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:41.657124 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:41.658179 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.658928 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.660876 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.661334 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.661477 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:41.662897 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:41.662904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:41.662908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:41.662930 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:41.662973 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:41.662989 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:41.663094 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:41.663321 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 00:57:41.663426 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:57:41.664891 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:41.654074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34797 (local address 127.0.97.2:37833)
0619 00:57:41.654979 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:41.655021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:41.656584 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:41.657355 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:41.657364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:41.657451 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:41.658009 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:41.658028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.659084 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.659096 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:41.660711 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:41.660723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:41.661569 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:41.661577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:41.661828 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:41.662560 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:41.662562 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:41.662604 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:41.662688 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:41.663093 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:41.663106 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:41.663117 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 00:57:41.663311 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 00:57:41.663765 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":274,"threads_started":1}
I20260619 00:57:41.666388 18424 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34797
I20260619 00:57:41.666690 18424 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:41.667431 18424 heartbeater.cc:511] Master 127.0.97.62:34797 requested a full tablet report, sending...
I20260619 00:57:41.668947 18107 ts_manager.cc:194] Registered new tserver with Master: 51fa655e2701442e9db7900bb2eb74d3 (127.0.97.2:37785)
I20260619 00:57:41.669772 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37833
W20260619 00:57:42.005327 18427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:42.005640 18427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:42.005777 18427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:42.013528 18427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:42.013652 18427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:42.013823 18427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:42.023499 18427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:42.025014 18427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:42.026962 18427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:42.041215 18435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:42.041291 18433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:42.041265 18432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:42.042234 18427 server_base.cc:1034] running on GCE node
I20260619 00:57:42.042968 18427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:42.043991 18427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:42.045264 18427 hybrid_clock.cc:648] HybridClock initialized: now 1781830662045210 us; error 46 us; skew 500 ppm
I20260619 00:57:42.045629 18427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:42.048025 18427 webserver.cc:460] Webserver started at http://127.0.97.3:44391/ using document root <none> and password file <none>
I20260619 00:57:42.048735 18427 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:42.048902 18427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:42.049238 18427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:42.052225 18427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "54979d0e0def41c096257e850ec9cb30"
format_stamp: "Formatted at 2026-06-19 00:57:42 on dist-test-slave-r7bh"
server_key: "7768ada3ff9673a38acdbd0ef92ca308"
server_key_iv: "4950cdafe9478b50cba88141e3402d76"
server_key_version: "encryptionkey@0"
I20260619 00:57:42.053143 18427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "54979d0e0def41c096257e850ec9cb30"
format_stamp: "Formatted at 2026-06-19 00:57:42 on dist-test-slave-r7bh"
server_key: "7768ada3ff9673a38acdbd0ef92ca308"
server_key_iv: "4950cdafe9478b50cba88141e3402d76"
server_key_version: "encryptionkey@0"
I20260619 00:57:42.058686 18427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:57:42.062695 18441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:42.064251 18427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:57:42.064457 18427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "54979d0e0def41c096257e850ec9cb30"
format_stamp: "Formatted at 2026-06-19 00:57:42 on dist-test-slave-r7bh"
server_key: "7768ada3ff9673a38acdbd0ef92ca308"
server_key_iv: "4950cdafe9478b50cba88141e3402d76"
server_key_version: "encryptionkey@0"
I20260619 00:57:42.064707 18427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:42.087349 18427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:42.088249 18427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:42.088613 18427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:42.089686 18427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:42.091380 18427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:42.091502 18427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:42.091636 18427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:42.091722 18427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:42.125066 18427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40985
I20260619 00:57:42.125159 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40985 every 8 connection(s)
I20260619 00:57:42.126446 18427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:42.129715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18427
I20260619 00:57:42.129910 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:42.130550 388 external_mini_cluster.cc:1428] Setting key 5d428789d5bc5989a0e79724d3068922
I20260619 00:57:42.140156 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.129083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48419 (local address 127.0.97.62:34797)
0619 00:57:42.129260 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.129265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.131104 (+ 1839us) server_negotiation.cc:407] Connection header received
0619 00:57:42.133619 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.133624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.133724 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.133827 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:42.134923 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.135689 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.137407 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.137838 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.137970 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.139329 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:42.139335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:42.139341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:42.139365 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:42.139394 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:42.139406 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:42.139501 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.139717 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.139822 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:57:42.141270 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.129815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34797 (local address 127.0.97.3:48419)
0619 00:57:42.130892 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.130935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.133303 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.134094 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.134103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.134207 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:42.134764 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.134779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.135839 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.135845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.137263 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.137275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.138059 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.138067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.138309 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.139011 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:42.139013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:42.139053 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:42.139134 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:42.139499 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:42.139513 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:42.139523 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.139704 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.140102 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
I20260619 00:57:42.142611 18554 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34797
I20260619 00:57:42.142935 18554 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:42.143698 18554 heartbeater.cc:511] Master 127.0.97.62:34797 requested a full tablet report, sending...
I20260619 00:57:42.145198 18107 ts_manager.cc:194] Registered new tserver with Master: 54979d0e0def41c096257e850ec9cb30 (127.0.97.3:40985)
I20260619 00:57:42.145915 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:42.146286 18107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48419
I20260619 00:57:42.155993 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.150308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.97.62:34797)
0619 00:57:42.150515 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.150519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.150928 (+ 409us) server_negotiation.cc:407] Connection header received
0619 00:57:42.151028 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.151033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.151107 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.151213 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:42.151665 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.152336 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.153758 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.154280 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.154404 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.155421 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:42.155427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:42.155432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:42.155453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:42.155502 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:42.155517 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:42.155624 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.155707 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.155777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 00:57:42.159480 18107 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:42.166579 18107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:42.207888 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.179714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.0.97.2:37785)
0619 00:57:42.198215 (+ 18501us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.198229 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.199249 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 00:57:42.199405 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.199416 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.199721 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.199934 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:42.200637 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.202274 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.205189 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.206687 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.206831 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.207119 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.207353 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.207525 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18266,"thread_start_us":19008,"threads_started":1}
I20260619 00:57:42.207902 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.179410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37785 (local address 127.0.0.1:54514)
0619 00:57:42.198716 (+ 19306us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.198740 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.198897 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.200041 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.200048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.200078 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:42.200422 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.200434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.202472 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.202478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.204976 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.204994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.206887 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.206893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.207108 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.207169 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.207308 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.207520 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19133,"mutex_wait_us":60,"thread_start_us":19594,"threads_started":1}
I20260619 00:57:42.209172 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.195589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41189 (local address 127.0.0.1:37006)
0619 00:57:42.199602 (+ 4013us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.199619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.199716 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.201761 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.201769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.201809 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:42.202261 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.202272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.204301 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.204308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.206453 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.206469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.208356 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.208363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.208513 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.208553 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.208650 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.208786 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3910,"mutex_wait_us":43,"thread_start_us":4167,"threads_started":1}
I20260619 00:57:42.210041 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.179133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40985 (local address 127.0.0.1:45594)
0619 00:57:42.194509 (+ 15376us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.195893 (+ 1384us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.196089 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.203487 (+ 7398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.203491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.203516 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:42.203913 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.203923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.205635 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.205641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.207464 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.207480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.209243 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.209249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.209428 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.209493 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.209567 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.209710 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14986,"mutex_wait_us":38,"thread_start_us":15420,"threads_started":1}
I20260619 00:57:42.210388 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.179172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.0.97.3:40985)
0619 00:57:42.202796 (+ 23624us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.202804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.202841 (+ 37us) server_negotiation.cc:407] Connection header received
0619 00:57:42.202930 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.202939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.203164 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.203341 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:42.204111 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.205498 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.207685 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.209049 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.209239 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.209675 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.209740 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.209901 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23091,"thread_start_us":22945,"threads_started":1}
I20260619 00:57:42.211051 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.195941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.97.1:41189)
0619 00:57:42.201120 (+ 5179us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.201129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.201160 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:57:42.201287 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.201297 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.201541 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.201743 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:42.202538 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.204173 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.206670 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.208161 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.208294 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.208560 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.210506 (+ 1946us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.210710 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5001,"thread_start_us":4859,"threads_started":1}
I20260619 00:57:42.212466 18293 heartbeater.cc:503] Master 127.0.97.62:34797 was elected leader, sending a full tablet report...
I20260619 00:57:42.212592 18359 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae (DEFAULT_TABLE table=test-table [id=b15730928823415c9d074741288088e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:42.214179 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f9710ca1b49ad8ff8f945100ebdae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:42.213722 18489 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae (DEFAULT_TABLE table=test-table [id=b15730928823415c9d074741288088e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:42.214942 18489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f9710ca1b49ad8ff8f945100ebdae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:42.222503 18228 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae (DEFAULT_TABLE table=test-table [id=b15730928823415c9d074741288088e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:42.224097 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f9710ca1b49ad8ff8f945100ebdae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:42.226241 18572 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Bootstrap starting.
I20260619 00:57:42.227566 18573 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Bootstrap starting.
I20260619 00:57:42.230901 18572 tablet_bootstrap.cc:654] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:42.231010 18573 tablet_bootstrap.cc:654] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:42.232232 18572 log.cc:824] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:42.232225 18573 log.cc:824] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:42.235404 18572 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: No bootstrap required, opened a new log
I20260619 00:57:42.235702 18572 ts_tablet_manager.cc:1397] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 00:57:42.236155 18573 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: No bootstrap required, opened a new log
I20260619 00:57:42.236433 18573 ts_tablet_manager.cc:1397] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 00:57:42.237963 18576 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Bootstrap starting.
I20260619 00:57:42.241080 18576 tablet_bootstrap.cc:654] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:42.242086 18576 log.cc:824] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:42.246165 18576 tablet_bootstrap.cc:492] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: No bootstrap required, opened a new log
I20260619 00:57:42.246447 18576 ts_tablet_manager.cc:1397] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260619 00:57:42.246584 18572 raft_consensus.cc:348] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.247061 18572 raft_consensus.cc:374] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:42.247159 18572 raft_consensus.cc:729] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54979d0e0def41c096257e850ec9cb30, State: Initialized, Role: FOLLOWER
I20260619 00:57:42.246938 18573 raft_consensus.cc:348] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.247421 18573 raft_consensus.cc:374] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:42.247545 18573 raft_consensus.cc:729] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa655e2701442e9db7900bb2eb74d3, State: Initialized, Role: FOLLOWER
I20260619 00:57:42.247824 18572 consensus_queue.cc:260] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.248142 18573 consensus_queue.cc:260] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.249300 18554 heartbeater.cc:503] Master 127.0.97.62:34797 was elected leader, sending a full tablet report...
I20260619 00:57:42.250224 18572 ts_tablet_manager.cc:1428] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260619 00:57:42.251686 18573 ts_tablet_manager.cc:1428] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260619 00:57:42.252087 18424 heartbeater.cc:503] Master 127.0.97.62:34797 was elected leader, sending a full tablet report...
I20260619 00:57:42.257064 18576 raft_consensus.cc:348] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.257400 18576 raft_consensus.cc:374] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:42.257493 18576 raft_consensus.cc:729] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab2e97ab7914b5bad4e94f4979b38e1, State: Initialized, Role: FOLLOWER
I20260619 00:57:42.257944 18576 consensus_queue.cc:260] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.259544 18576 ts_tablet_manager.cc:1428] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 00:57:42.324879 18579 raft_consensus.cc:482] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:42.325289 18579 raft_consensus.cc:504] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.327575 18579 leader_election.cc:288] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54979d0e0def41c096257e850ec9cb30 (127.0.97.3:40985), 3ab2e97ab7914b5bad4e94f4979b38e1 (127.0.97.1:41189)
I20260619 00:57:42.335438 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.328248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55633 (local address 127.0.97.3:40985)
0619 00:57:42.328510 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.328519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.329190 (+ 671us) server_negotiation.cc:407] Connection header received
0619 00:57:42.329476 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.329482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.329557 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.329667 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:42.330303 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.331282 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.333512 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.334480 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.334577 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.334651 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.335041 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.335179 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:57:42.335543 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.328095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40985 (local address 127.0.97.2:55633)
0619 00:57:42.329042 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.329090 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.329317 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.329684 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.329688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.329728 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:42.330139 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.330152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.331423 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.331429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.333303 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.333318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.334683 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.334690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.334863 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.334914 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.335001 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.335140 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":2,"thread_start_us":584,"threads_started":1}
I20260619 00:57:42.335801 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.329617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57921 (local address 127.0.97.1:41189)
0619 00:57:42.329839 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.329844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.329864 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:57:42.330067 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.330075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.330172 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.330263 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:42.330849 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.331727 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.333942 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.334827 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.334952 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.335059 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.335386 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.335547 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:57:42.336035 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.329131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41189 (local address 127.0.97.2:57921)
0619 00:57:42.329727 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:42.329746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:42.329895 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:42.330294 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:42.330298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:42.330322 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:42.330698 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.330711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.331864 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.331867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.333699 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:42.333712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.335034 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.335040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:42.335211 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.335241 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:57:42.335332 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 00:57:42.335455 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":19,"thread_start_us":732,"threads_started":1}
I20260619 00:57:42.336714 18509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f9710ca1b49ad8ff8f945100ebdae" candidate_uuid: "51fa655e2701442e9db7900bb2eb74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54979d0e0def41c096257e850ec9cb30" is_pre_election: true
I20260619 00:57:42.337123 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f9710ca1b49ad8ff8f945100ebdae" candidate_uuid: "51fa655e2701442e9db7900bb2eb74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" is_pre_election: true
I20260619 00:57:42.337296 18509 raft_consensus.cc:2454] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa655e2701442e9db7900bb2eb74d3 in term 0.
I20260619 00:57:42.337617 18248 raft_consensus.cc:2454] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa655e2701442e9db7900bb2eb74d3 in term 0.
I20260619 00:57:42.337994 18315 leader_election.cc:302] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51fa655e2701442e9db7900bb2eb74d3, 54979d0e0def41c096257e850ec9cb30; no voters:
I20260619 00:57:42.338413 18579 raft_consensus.cc:2781] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:42.338562 18579 raft_consensus.cc:482] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:42.338691 18579 raft_consensus.cc:3037] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:42.341041 18579 raft_consensus.cc:504] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.341614 18579 leader_election.cc:288] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [CANDIDATE]: Term 1 election: Requested vote from peers 54979d0e0def41c096257e850ec9cb30 (127.0.97.3:40985), 3ab2e97ab7914b5bad4e94f4979b38e1 (127.0.97.1:41189)
I20260619 00:57:42.342166 18509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f9710ca1b49ad8ff8f945100ebdae" candidate_uuid: "51fa655e2701442e9db7900bb2eb74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54979d0e0def41c096257e850ec9cb30"
I20260619 00:57:42.342267 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f9710ca1b49ad8ff8f945100ebdae" candidate_uuid: "51fa655e2701442e9db7900bb2eb74d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1"
I20260619 00:57:42.342382 18509 raft_consensus.cc:3037] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:42.342521 18248 raft_consensus.cc:3037] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:42.344736 18509 raft_consensus.cc:2454] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa655e2701442e9db7900bb2eb74d3 in term 1.
I20260619 00:57:42.344751 18248 raft_consensus.cc:2454] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa655e2701442e9db7900bb2eb74d3 in term 1.
I20260619 00:57:42.345227 18315 leader_election.cc:302] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51fa655e2701442e9db7900bb2eb74d3, 54979d0e0def41c096257e850ec9cb30; no voters:
I20260619 00:57:42.345564 18579 raft_consensus.cc:2781] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:42.346015 18579 raft_consensus.cc:686] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 1 LEADER]: Becoming Leader. State: Replica: 51fa655e2701442e9db7900bb2eb74d3, State: Running, Role: LEADER
I20260619 00:57:42.346513 18579 consensus_queue.cc:237] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } }
I20260619 00:57:42.351281 18107 catalog_manager.cc:5495] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51fa655e2701442e9db7900bb2eb74d3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "51fa655e2701442e9db7900bb2eb74d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51fa655e2701442e9db7900bb2eb74d3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37785 } health_report { overall_health: HEALTHY } } }
W20260619 00:57:42.378248 18555 tablet.cc:2369] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:42.383677 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.378058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.0.97.2:37785)
0619 00:57:42.378276 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.378285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.378425 (+ 140us) server_negotiation.cc:407] Connection header received
0619 00:57:42.378633 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.378637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.378751 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.378876 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:42.379323 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.380036 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.381510 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.381946 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.382060 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.382521 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:42.383058 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:42.383178 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.383286 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.383355 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:57:42.392565 18248 raft_consensus.cc:1261] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 1 FOLLOWER]: Refusing update from remote peer 51fa655e2701442e9db7900bb2eb74d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:42.392565 18509 raft_consensus.cc:1261] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 1 FOLLOWER]: Refusing update from remote peer 51fa655e2701442e9db7900bb2eb74d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:42.393649 18583 consensus_queue.cc:1035] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54979d0e0def41c096257e850ec9cb30" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:42.394019 18579 consensus_queue.cc:1035] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab2e97ab7914b5bad4e94f4979b38e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:57:42.403136 18425 tablet.cc:2369] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:42.418437 18587 mvcc.cc:205] Tried to move back new op lower bound from 7298378393148116992 to 7298378392975691776. Current Snapshot: MvccSnapshot[applied={T|T < 7298378393148116992}]
I20260619 00:57:42.421752 18107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:42.422003 18107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:42.425550 18107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f9ca98943ff4ff4bfa55e513cce8844: Sending DeleteTablet for 3 replicas of tablet ea8f9710ca1b49ad8ff8f945100ebdae
I20260619 00:57:42.426918 18228 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:42 UTC) from {username='slave'} at 127.0.0.1:37006
I20260619 00:57:42.426939 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:42 UTC) from {username='slave'} at 127.0.0.1:54514
I20260619 00:57:42.427896 18596 tablet_replica.cc:330] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: stopping tablet replica
I20260619 00:57:42.428433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18166
I20260619 00:57:42.428447 18596 raft_consensus.cc:2229] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:42.429005 18596 raft_consensus.cc:2258] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:42.429847 18597 tablet_replica.cc:330] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: stopping tablet replica
I20260619 00:57:42.430271 18597 raft_consensus.cc:2229] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:42.430980 18597 raft_consensus.cc:2258] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:42.432305 18489 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f9710ca1b49ad8ff8f945100ebdae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:42 UTC) from {username='slave'} at 127.0.0.1:45594
I20260619 00:57:42.434108 18596 ts_tablet_manager.cc:1905] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:42.440163 18596 ts_tablet_manager.cc:1918] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:42.440371 18596 log.cc:1196] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/ea8f9710ca1b49ad8ff8f945100ebdae
I20260619 00:57:42.440982 18599 tablet_replica.cc:330] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: stopping tablet replica
I20260619 00:57:42.441067 18596 ts_tablet_manager.cc:1939] T ea8f9710ca1b49ad8ff8f945100ebdae P 3ab2e97ab7914b5bad4e94f4979b38e1: Deleting consensus metadata
I20260619 00:57:42.441481 18599 raft_consensus.cc:2229] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:42.442075 18599 raft_consensus.cc:2258] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:42.443147 18094 catalog_manager.cc:4841] TS 3ab2e97ab7914b5bad4e94f4979b38e1 (127.0.97.1:41189): tablet ea8f9710ca1b49ad8ff8f945100ebdae (table test-table [id=b15730928823415c9d074741288088e7]) successfully deleted
I20260619 00:57:42.444876 18599 ts_tablet_manager.cc:1905] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:42.445547 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.430568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.0.97.1:41189)
0619 00:57:42.430761 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.430767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.430787 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:57:42.430852 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.430857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.430959 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.431063 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:42.431684 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.432802 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.434739 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.435399 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.435499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.443548 (+ 8049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:42.443560 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:42.443579 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:42.444178 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:42.444232 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:42.444255 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:42.444411 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.445048 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.445179 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:57:42.445531 18597 ts_tablet_manager.cc:1905] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:42.446300 18288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:42.453153 18599 ts_tablet_manager.cc:1918] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:42.453456 18599 log.cc:1196] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/ea8f9710ca1b49ad8ff8f945100ebdae
I20260619 00:57:42.454130 18599 ts_tablet_manager.cc:1939] T ea8f9710ca1b49ad8ff8f945100ebdae P 54979d0e0def41c096257e850ec9cb30: Deleting consensus metadata
I20260619 00:57:42.454213 18597 ts_tablet_manager.cc:1918] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:42.454499 18597 log.cc:1196] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/ea8f9710ca1b49ad8ff8f945100ebdae
I20260619 00:57:42.455125 18597 ts_tablet_manager.cc:1939] T ea8f9710ca1b49ad8ff8f945100ebdae P 51fa655e2701442e9db7900bb2eb74d3: Deleting consensus metadata
I20260619 00:57:42.456274 18094 catalog_manager.cc:4841] TS 54979d0e0def41c096257e850ec9cb30 (127.0.97.3:40985): tablet ea8f9710ca1b49ad8ff8f945100ebdae (table test-table [id=b15730928823415c9d074741288088e7]) successfully deleted
I20260619 00:57:42.457201 18091 catalog_manager.cc:4841] TS 51fa655e2701442e9db7900bb2eb74d3 (127.0.97.2:37785): tablet ea8f9710ca1b49ad8ff8f945100ebdae (table test-table [id=b15730928823415c9d074741288088e7]) successfully deleted
I20260619 00:57:42.546033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18166
I20260619 00:57:42.572072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18297
I20260619 00:57:42.579437 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.572938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.0.97.2:37785)
0619 00:57:42.573169 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.573175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.573196 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:57:42.573391 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.573397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.573495 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.573626 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:42.574041 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.574875 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.576215 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.576704 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.576874 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.577955 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:42.577964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:42.577979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:42.578634 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:42.578686 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:42.578705 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:42.578821 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.578965 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.579078 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:57:42.580307 18419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:42.693852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18297
I20260619 00:57:42.713860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18427
I20260619 00:57:42.720468 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:42.714645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.97.3:40985)
0619 00:57:42.714818 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:42.714823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:42.714841 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:57:42.714936 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:42.714939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:42.715031 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:42.715119 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:42.715518 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.716253 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.717590 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:42.717999 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:42.718112 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:42.719162 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:42.719172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:42.719187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:42.719781 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:42.719837 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:42.719859 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:42.719950 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:42.720121 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 00:57:42.720224 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:57:42.721225 18549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:42.814370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18427
I20260619 00:57:42.831395 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18076
I20260619 00:57:42.832559 18137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:42.943764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18076
2026-06-19T00:57:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2916 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:57:43 dist-test-slave-r7bh krb5kdc[18606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:57:43 dist-test-slave-r7bh krb5kdc[18606](info): set up 2 sockets
Jun 19 00:57:43 dist-test-slave-r7bh krb5kdc[18606](info): commencing operation
krb5kdc: starting...
W20260619 00:57:44.061282 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:57:44 dist-test-slave-r7bh krb5kdc[18606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:57:44.310375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34021
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:44.645365 18619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:44.645709 18619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:44.645882 18619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:44.653261 18619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:44.653416 18619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:44.653532 18619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:44.653641 18619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:44.653723 18619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:44.663676 18619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:44.665089 18619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:44.666420 18619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:44.678037 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:44.678071 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:44.678097 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:44.679375 18619 server_base.cc:1034] running on GCE node
I20260619 00:57:44.680191 18619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:44.681685 18619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:44.682924 18619 hybrid_clock.cc:648] HybridClock initialized: now 1781830664682881 us; error 51 us; skew 500 ppm
I20260619 00:57:44.683271 18619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:44 dist-test-slave-r7bh krb5kdc[18606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:44.686524 18619 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:57:44.688038 18619 webserver.cc:460] Webserver started at http://127.0.97.62:45635/ using document root <none> and password file <none>
I20260619 00:57:44.688753 18619 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:44.688925 18619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:44.689252 18619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:44.692211 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "c6dadb13e16c4ad5831e49115e295859"
format_stamp: "Formatted at 2026-06-19 00:57:44 on dist-test-slave-r7bh"
server_key: "d605b993aea42bfaa5009b4b67e0731f"
server_key_iv: "7bbd2923e2eecc96a0c1d142961ad58c"
server_key_version: "encryptionkey@0"
I20260619 00:57:44.693027 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6dadb13e16c4ad5831e49115e295859"
format_stamp: "Formatted at 2026-06-19 00:57:44 on dist-test-slave-r7bh"
server_key: "d605b993aea42bfaa5009b4b67e0731f"
server_key_iv: "7bbd2923e2eecc96a0c1d142961ad58c"
server_key_version: "encryptionkey@0"
I20260619 00:57:44.698088 18619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 00:57:44.701498 18634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:44.702816 18619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:44.703003 18619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "c6dadb13e16c4ad5831e49115e295859"
format_stamp: "Formatted at 2026-06-19 00:57:44 on dist-test-slave-r7bh"
server_key: "d605b993aea42bfaa5009b4b67e0731f"
server_key_iv: "7bbd2923e2eecc96a0c1d142961ad58c"
server_key_version: "encryptionkey@0"
I20260619 00:57:44.703217 18619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:44.721442 18619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:44.726181 18619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:44.726498 18619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:44.745280 18619 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34021
I20260619 00:57:44.745299 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34021 every 8 connection(s)
I20260619 00:57:44.746773 18619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:44.750900 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:44.755477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18619
I20260619 00:57:44.755676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:44.756168 388 external_mini_cluster.cc:1428] Setting key fc2f93b9848e01d08f2ab1614dca5935
I20260619 00:57:44.763572 18686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: Bootstrap starting.
I20260619 00:57:44.767894 18686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:44.769385 18686 log.cc:824] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:44.773339 18686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: No bootstrap required, opened a new log
Jun 19 00:57:44 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:44.785509 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:44.758195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.0.97.62:34021)
0619 00:57:44.759066 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:44.759083 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:44.759137 (+ 54us) server_negotiation.cc:407] Connection header received
0619 00:57:44.761776 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:44.761807 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:44.762341 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:44.762795 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:44.764144 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:44.766194 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:44.768530 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:44.769624 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:44.769797 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:44.775629 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:44.775643 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:44.775663 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:44.775723 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:44.780482 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:44.781486 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:44.781495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:44.781504 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:44.781725 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:44.782180 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:44.782185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:44.782188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:44.782779 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:44.782988 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:44.783550 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 00:57:44.784124 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":355,"threads_started":1}
I20260619 00:57:44.785560 18686 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } }
I20260619 00:57:44.786041 18686 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:44.786172 18686 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dadb13e16c4ad5831e49115e295859, State: Initialized, Role: FOLLOWER
I20260619 00:57:44.786792 18686 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } }
I20260619 00:57:44.787086 18686 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:44.787245 18686 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:44.787442 18686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:44.790266 18686 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } }
I20260619 00:57:44.790719 18686 leader_election.cc:302] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6dadb13e16c4ad5831e49115e295859; no voters:
I20260619 00:57:44.791347 18686 leader_election.cc:288] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:44.791512 18691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:44.793076 18691 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [term 1 LEADER]: Becoming Leader. State: Replica: c6dadb13e16c4ad5831e49115e295859, State: Running, Role: LEADER
I20260619 00:57:44.793628 18686 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:44.793630 18691 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } }
I20260619 00:57:44.796718 18693 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6dadb13e16c4ad5831e49115e295859. Latest consensus state: current_term: 1 leader_uuid: "c6dadb13e16c4ad5831e49115e295859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } } }
I20260619 00:57:44.797207 18693 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:44.797875 18692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6dadb13e16c4ad5831e49115e295859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dadb13e16c4ad5831e49115e295859" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34021 } } }
I20260619 00:57:44.798173 18698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:44.798207 18692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:44.803725 18698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:44.823071 18698 catalog_manager.cc:1347] Generated new cluster ID: 8d659d20437a47a9bdcffdd26f8abdd7
I20260619 00:57:44.823268 18698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:44.842695 18698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:44.844609 18698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:44.860999 18698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: Generated new TSK 0
I20260619 00:57:44.861930 18698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:57:44.935508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:45.270080 18712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:45.270354 18712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:45.270541 18712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:45.277998 18712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:45.278102 18712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:45.278277 18712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:45.288340 18712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:45.289681 18712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:45.290992 18712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:45.304054 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:45.303960 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:45.304010 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:45.305188 18712 server_base.cc:1034] running on GCE node
I20260619 00:57:45.305868 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:45.306967 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:45.308229 18712 hybrid_clock.cc:648] HybridClock initialized: now 1781830665308165 us; error 44 us; skew 500 ppm
I20260619 00:57:45.308610 18712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:45 dist-test-slave-r7bh krb5kdc[18606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:45.312067 18712 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:57:45.313781 18712 webserver.cc:460] Webserver started at http://127.0.97.1:36501/ using document root <none> and password file <none>
I20260619 00:57:45.314527 18712 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:45.314678 18712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:45.315014 18712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:45.318143 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "99647dc7d1544e69ad5d3006624f890b"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "42dc0be28003bb56386f94073c1e171f"
server_key_iv: "15ba1356947821dd87a2e86700c115fd"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.318977 18712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "99647dc7d1544e69ad5d3006624f890b"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "42dc0be28003bb56386f94073c1e171f"
server_key_iv: "15ba1356947821dd87a2e86700c115fd"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.324445 18712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:57:45.328136 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.329658 18712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:57:45.329855 18712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "99647dc7d1544e69ad5d3006624f890b"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "42dc0be28003bb56386f94073c1e171f"
server_key_iv: "15ba1356947821dd87a2e86700c115fd"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.330081 18712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:45.363818 18712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:45.368221 18712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:45.368542 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:45.369817 18712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:45.371172 18712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:45.371276 18712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.371433 18712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:45.371506 18712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.404685 18712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35843
I20260619 00:57:45.404722 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35843 every 8 connection(s)
I20260619 00:57:45.406097 18712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:45.410604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18712
I20260619 00:57:45.410811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:45.411365 388 external_mini_cluster.cc:1428] Setting key 68f621c8aa29917c1245be2d16343d35
Jun 19 00:57:45 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:45.430722 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:45.408950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54575 (local address 127.0.97.62:34021)
0619 00:57:45.409460 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:45.409464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:45.410529 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 00:57:45.412938 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:45.412943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:45.413019 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:45.413136 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:45.415262 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.416002 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.418570 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.418957 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.419064 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:45.423488 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:45.423498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:45.423505 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:45.423558 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:45.427296 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:45.428070 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:45.428078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:45.428084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:45.428233 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:45.428651 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:45.428659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:45.428665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:45.429092 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:45.429236 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:45.429950 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 00:57:45.430221 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":297,"threads_started":1}
I20260619 00:57:45.431416 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:45.409509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34021 (local address 127.0.97.1:54575)
0619 00:57:45.410314 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:45.410384 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:45.412617 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:45.413370 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:45.413380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:45.414144 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:45.415014 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:45.415045 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.416185 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.416194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:45.418411 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:45.418429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.419223 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.419232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:45.419576 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:45.420389 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:45.420424 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:45.423234 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:45.427460 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:45.427467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:45.427481 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:45.427926 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:45.428338 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:45.428341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:45.428343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:45.428527 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:45.429223 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:45.429229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:45.429557 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 00:57:45.429783 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 00:57:45.430199 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":318,"threads_started":1}
I20260619 00:57:45.433151 18840 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34021
I20260619 00:57:45.433451 18840 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:45.434331 18840 heartbeater.cc:511] Master 127.0.97.62:34021 requested a full tablet report, sending...
I20260619 00:57:45.436939 18651 ts_manager.cc:194] Registered new tserver with Master: 99647dc7d1544e69ad5d3006624f890b (127.0.97.1:35843)
I20260619 00:57:45.439210 18651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54575
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:57:45.470332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:45.795840 18846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:45.796121 18846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:45.796290 18846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:45.803514 18846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:45.803617 18846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:45.803735 18846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:45.813709 18846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:45.814980 18846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:45.816216 18846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:45.829360 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:45.829489 18854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:45.829394 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:45.830286 18846 server_base.cc:1034] running on GCE node
I20260619 00:57:45.830837 18846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:45.831743 18846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:45.832908 18846 hybrid_clock.cc:648] HybridClock initialized: now 1781830665832831 us; error 59 us; skew 500 ppm
I20260619 00:57:45.833264 18846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:45 dist-test-slave-r7bh krb5kdc[18606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:45.836479 18846 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:57:45.838070 18846 webserver.cc:460] Webserver started at http://127.0.97.2:34541/ using document root <none> and password file <none>
I20260619 00:57:45.838773 18846 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:45.838919 18846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:45.839233 18846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:45.842239 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "54b0adac18069ed34ebb7f277b43d01b"
server_key_iv: "bdb81a265f62607e281ef4711632b241"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.843060 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "54b0adac18069ed34ebb7f277b43d01b"
server_key_iv: "bdb81a265f62607e281ef4711632b241"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.848294 18846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:57:45.851933 18861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.853471 18846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:45.853650 18846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d"
format_stamp: "Formatted at 2026-06-19 00:57:45 on dist-test-slave-r7bh"
server_key: "54b0adac18069ed34ebb7f277b43d01b"
server_key_iv: "bdb81a265f62607e281ef4711632b241"
server_key_version: "encryptionkey@0"
I20260619 00:57:45.853880 18846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:45.886212 18846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:45.890576 18846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:45.890897 18846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:45.891906 18846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:45.893304 18846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:45.893412 18846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.893582 18846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:45.893654 18846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:45.926939 18846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41445
I20260619 00:57:45.926985 18973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41445 every 8 connection(s)
I20260619 00:57:45.928261 18846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:45.934201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18846
I20260619 00:57:45.934386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:45.934942 388 external_mini_cluster.cc:1428] Setting key 7e9a8786322cb4f96491550d5169fa31
Jun 19 00:57:45 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:45.948414 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:45.930783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55997 (local address 127.0.97.62:34021)
0619 00:57:45.930995 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:45.931001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:45.932285 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 00:57:45.934275 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:45.934282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:45.934387 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:45.934510 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:45.936099 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.936882 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.938413 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.938847 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.938982 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:45.942686 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:45.942691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:45.942695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:45.942727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:45.945413 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:45.946136 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:45.946145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:45.946149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:45.946277 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:45.946703 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:45.946709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:45.946713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:45.946978 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:45.947135 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:45.947833 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 00:57:45.948065 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:57:45.949512 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:45.931392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34021 (local address 127.0.97.2:55997)
0619 00:57:45.932124 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:45.932161 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:45.933979 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:45.934782 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:45.934791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:45.935361 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:45.935919 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:45.935937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.937050 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.937054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:45.938262 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:45.938273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:45.939078 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:45.939085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:45.939306 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:45.939991 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:45.940017 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:45.942444 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:45.945561 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:45.945571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:45.945588 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:45.946005 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:45.946387 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:45.946393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:45.946397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:45.946591 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:45.947108 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:45.947117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:45.947459 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 00:57:45.947736 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 00:57:45.948286 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260619 00:57:45.951016 18974 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34021
I20260619 00:57:45.951329 18974 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:45.952049 18974 heartbeater.cc:511] Master 127.0.97.62:34021 requested a full tablet report, sending...
I20260619 00:57:45.953586 18651 ts_manager.cc:194] Registered new tserver with Master: 0f2bb1d6fa1944d4ac6a0ab333766d5d (127.0.97.2:41445)
I20260619 00:57:45.954905 18651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55997
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:57:45.990255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:46.321240 18979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:46.321532 18979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:46.321687 18979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:46.329017 18979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:46.329134 18979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:46.329308 18979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:46.339486 18979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:46.341285 18979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:46.343277 18979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:46.357610 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:46.357570 18987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:46.357534 18985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:46.359297 18979 server_base.cc:1034] running on GCE node
I20260619 00:57:46.360013 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:46.361150 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:46.362457 18979 hybrid_clock.cc:648] HybridClock initialized: now 1781830666362363 us; error 35 us; skew 500 ppm
I20260619 00:57:46.362967 18979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:46 dist-test-slave-r7bh krb5kdc[18606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:46.366513 18979 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:57:46.368021 18979 webserver.cc:460] Webserver started at http://127.0.97.3:32949/ using document root <none> and password file <none>
I20260619 00:57:46.368743 18979 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:46.368918 18979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:46.369253 18979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:46.372275 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "82131fb9854b45e7ba2eecbf92d5ec18"
format_stamp: "Formatted at 2026-06-19 00:57:46 on dist-test-slave-r7bh"
server_key: "dec989105cbb88fe6ef54f9c75b5dac4"
server_key_iv: "8616c65a66a656e6b7fcfc92d4b6c8bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:46.373205 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "82131fb9854b45e7ba2eecbf92d5ec18"
format_stamp: "Formatted at 2026-06-19 00:57:46 on dist-test-slave-r7bh"
server_key: "dec989105cbb88fe6ef54f9c75b5dac4"
server_key_iv: "8616c65a66a656e6b7fcfc92d4b6c8bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:46.378350 18979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:57:46.381765 18994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:46.383021 18979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:46.383201 18979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "82131fb9854b45e7ba2eecbf92d5ec18"
format_stamp: "Formatted at 2026-06-19 00:57:46 on dist-test-slave-r7bh"
server_key: "dec989105cbb88fe6ef54f9c75b5dac4"
server_key_iv: "8616c65a66a656e6b7fcfc92d4b6c8bf"
server_key_version: "encryptionkey@0"
I20260619 00:57:46.383401 18979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:46.402563 18979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:46.408370 18979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:46.408716 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:46.409977 18979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:46.411772 18979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:46.411875 18979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:46.411984 18979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:46.412029 18979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:46.442574 18840 heartbeater.cc:503] Master 127.0.97.62:34021 was elected leader, sending a full tablet report...
I20260619 00:57:46.445524 18979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39425
I20260619 00:57:46.445564 19106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39425 every 8 connection(s)
I20260619 00:57:46.446800 18979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:46.454815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 18979
I20260619 00:57:46.454995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:46.455394 388 external_mini_cluster.cc:1428] Setting key f4e3a33a7691a2d444df65b65f9ff0ee
Jun 19 00:57:46 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:46.466958 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.449347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41343 (local address 127.0.97.62:34021)
0619 00:57:46.449834 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.449838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.450701 (+ 863us) server_negotiation.cc:407] Connection header received
0619 00:57:46.452507 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.452513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.452607 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.452724 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:46.454213 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.455113 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.457136 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.457967 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.458126 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.461707 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:46.461712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:46.461715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:46.461758 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:46.464238 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.464984 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.464988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.464990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.465128 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.465506 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.465509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.465511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.465760 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:46.465866 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.466522 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.466684 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260619 00:57:46.467933 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.449836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34021 (local address 127.0.97.3:41343)
0619 00:57:46.450548 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.450591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.452249 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.452989 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.452997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.453501 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:46.454037 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.454053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.455271 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.455275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.456941 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.456970 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.458308 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.458317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.458590 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.459281 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:46.459308 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:46.461505 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:46.464372 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:46.464378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:46.464392 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:46.464871 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:46.465225 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:46.465228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:46.465231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:46.465410 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:46.465860 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:46.465866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:46.466202 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.466427 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.466810 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":274,"threads_started":1}
I20260619 00:57:46.469635 19107 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34021
I20260619 00:57:46.469960 19107 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:46.470908 19107 heartbeater.cc:511] Master 127.0.97.62:34021 requested a full tablet report, sending...
I20260619 00:57:46.472381 18651 ts_manager.cc:194] Registered new tserver with Master: 82131fb9854b45e7ba2eecbf92d5ec18 (127.0.97.3:39425)
I20260619 00:57:46.473284 18651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41343
I20260619 00:57:46.483186 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:46.499939 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.487279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.0.97.62:34021)
0619 00:57:46.487505 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.487515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.487926 (+ 411us) server_negotiation.cc:407] Connection header received
0619 00:57:46.488055 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.488061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.488145 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.488225 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:46.489685 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.490392 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.491800 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.492182 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.492274 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.494950 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:46.494955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:46.494958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:46.494991 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:46.497451 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.498155 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.498159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.498161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.498270 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.498624 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.498627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.498629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.498914 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:46.499107 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.499397 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.499534 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:57:46.503881 18651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:46.511541 18651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:46.535058 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.525206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.97.1:35843)
0619 00:57:46.525852 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.525864 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.525895 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:57:46.526108 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.526118 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.526401 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.526656 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:46.528035 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.529813 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.532242 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.533684 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.533829 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.534100 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.534510 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.534692 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":315,"threads_started":1}
I20260619 00:57:46.535164 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.524846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35843 (local address 127.0.0.1:50174)
0619 00:57:46.525590 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.525645 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.525827 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.527020 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.527031 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.527076 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:46.527713 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.527741 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.529976 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.529982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.532080 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.532093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.533972 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.533979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.534230 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.534272 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.534462 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.534687 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":31,"thread_start_us":303,"threads_started":1}
I20260619 00:57:46.537655 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.528083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39425 (local address 127.0.0.1:50830)
0619 00:57:46.528905 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.528924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.529030 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.530204 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.530209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.530236 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:46.530729 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.530742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.532655 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.532660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.535191 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.535206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.536743 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.536752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.536963 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.536997 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.537069 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.537199 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":48,"thread_start_us":885,"threads_started":1}
I20260619 00:57:46.538482 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.525965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41445 (local address 127.0.0.1:57560)
0619 00:57:46.527968 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.527986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.528097 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.530481 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.530486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.530509 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:46.531062 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.531084 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.533002 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.533007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.535674 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.535688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.537761 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.537766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.537943 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.537974 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.538060 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.538198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"mutex_wait_us":65,"thread_start_us":1686,"threads_started":1}
I20260619 00:57:46.538938 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.528700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.0.97.3:39425)
0619 00:57:46.529576 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.529584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.529615 (+ 31us) server_negotiation.cc:407] Connection header received
0619 00:57:46.529700 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.529707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.529978 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.530199 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:46.530918 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.532506 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.535383 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.536496 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.538115 (+ 1619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.538335 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.538405 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.538578 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":562,"threads_started":1}
I20260619 00:57:46.539139 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.526270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.97.2:41445)
0619 00:57:46.528605 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.528614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.528641 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:57:46.528736 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.528745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.529987 (+ 1242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.530218 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:46.531281 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.532781 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.535845 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.537265 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.537394 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.537618 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.538728 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.538870 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165,"thread_start_us":2048,"threads_started":1}
I20260619 00:57:46.539963 18775 tablet_service.cc:1467] Processing CreateTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c (DEFAULT_TABLE table=test-table [id=cdec70d7068b466da8f482b6d27566ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:46.541352 18775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9821c7d7f86d41d7b2aacd1730741d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:46.542635 18909 tablet_service.cc:1467] Processing CreateTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c (DEFAULT_TABLE table=test-table [id=cdec70d7068b466da8f482b6d27566ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:46.542635 19042 tablet_service.cc:1467] Processing CreateTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c (DEFAULT_TABLE table=test-table [id=cdec70d7068b466da8f482b6d27566ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:46.544011 19042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9821c7d7f86d41d7b2aacd1730741d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:46.544013 18909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9821c7d7f86d41d7b2aacd1730741d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:46.553431 19126 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Bootstrap starting.
I20260619 00:57:46.557211 19126 tablet_bootstrap.cc:654] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:46.558470 19126 log.cc:824] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:46.559984 19128 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Bootstrap starting.
I20260619 00:57:46.560863 19127 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Bootstrap starting.
I20260619 00:57:46.561928 19126 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: No bootstrap required, opened a new log
I20260619 00:57:46.562216 19126 ts_tablet_manager.cc:1397] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 00:57:46.564592 19127 tablet_bootstrap.cc:654] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:46.565904 19127 log.cc:824] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:46.569144 19128 tablet_bootstrap.cc:654] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:46.569360 19127 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: No bootstrap required, opened a new log
I20260619 00:57:46.569646 19127 ts_tablet_manager.cc:1397] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 00:57:46.570258 19128 log.cc:824] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:46.572966 19128 tablet_bootstrap.cc:492] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: No bootstrap required, opened a new log
I20260619 00:57:46.573235 19128 ts_tablet_manager.cc:1397] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260619 00:57:46.573311 19126 raft_consensus.cc:348] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.573767 19126 raft_consensus.cc:374] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:46.573899 19126 raft_consensus.cc:729] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99647dc7d1544e69ad5d3006624f890b, State: Initialized, Role: FOLLOWER
I20260619 00:57:46.574544 19126 consensus_queue.cc:260] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.578437 19126 ts_tablet_manager.cc:1428] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260619 00:57:46.579751 19127 raft_consensus.cc:348] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.580220 19127 raft_consensus.cc:374] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:46.580327 19127 raft_consensus.cc:729] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82131fb9854b45e7ba2eecbf92d5ec18, State: Initialized, Role: FOLLOWER
I20260619 00:57:46.581028 19127 consensus_queue.cc:260] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.582155 19107 heartbeater.cc:503] Master 127.0.97.62:34021 was elected leader, sending a full tablet report...
I20260619 00:57:46.582863 19127 ts_tablet_manager.cc:1428] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 00:57:46.583101 19128 raft_consensus.cc:348] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.583570 19128 raft_consensus.cc:374] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:46.583714 19128 raft_consensus.cc:729] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2bb1d6fa1944d4ac6a0ab333766d5d, State: Initialized, Role: FOLLOWER
I20260619 00:57:46.584401 19128 consensus_queue.cc:260] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.585623 18974 heartbeater.cc:503] Master 127.0.97.62:34021 was elected leader, sending a full tablet report...
I20260619 00:57:46.586313 19128 ts_tablet_manager.cc:1428] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 00:57:46.658607 18841 tablet.cc:2369] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:46.680436 18975 tablet.cc:2369] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:46.698482 19108 tablet.cc:2369] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:46.721930 19133 raft_consensus.cc:482] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:46.722271 19133 raft_consensus.cc:504] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.724005 19133 leader_election.cc:288] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99647dc7d1544e69ad5d3006624f890b (127.0.97.1:35843), 0f2bb1d6fa1944d4ac6a0ab333766d5d (127.0.97.2:41445)
I20260619 00:57:46.730487 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.724451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53803 (local address 127.0.97.2:41445)
0619 00:57:46.724656 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.724663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.724794 (+ 131us) server_negotiation.cc:407] Connection header received
0619 00:57:46.725061 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.725068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.725151 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.725263 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:46.725757 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.726687 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.728508 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.729610 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.729713 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.729785 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.730138 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.730233 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:57:46.730614 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.724327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41445 (local address 127.0.97.3:53803)
0619 00:57:46.724658 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.724684 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.724889 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.725293 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.725297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.725319 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:46.725589 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.725602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.726828 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.726833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.728346 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.728359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.729779 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.729786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.729978 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.730012 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.730117 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.730256 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 00:57:46.731317 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.724451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36153 (local address 127.0.97.1:35843)
0619 00:57:46.724656 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.724663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.725387 (+ 724us) server_negotiation.cc:407] Connection header received
0619 00:57:46.725572 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.725577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.725667 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.725764 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:46.726334 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.727269 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.729262 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.730263 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.730392 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.730495 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.730884 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.731040 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:57:46.731976 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.724399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35843 (local address 127.0.97.3:36153)
0619 00:57:46.725259 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:46.725277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:46.725417 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:46.725783 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:46.725787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:46.725815 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:46.726152 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.726164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.727431 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.727435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.729106 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:46.729121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.730478 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.730487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:46.730702 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.730735 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:57:46.731510 (+ 775us) client_negotiation.cc:240] Negotiation successful
0619 00:57:46.731656 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":475,"threads_started":1}
I20260619 00:57:46.731779 18929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9821c7d7f86d41d7b2aacd1730741d3c" candidate_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" is_pre_election: true
I20260619 00:57:46.732372 18929 raft_consensus.cc:2454] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82131fb9854b45e7ba2eecbf92d5ec18 in term 0.
I20260619 00:57:46.733081 18996 leader_election.cc:302] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2bb1d6fa1944d4ac6a0ab333766d5d, 82131fb9854b45e7ba2eecbf92d5ec18; no voters:
I20260619 00:57:46.732988 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9821c7d7f86d41d7b2aacd1730741d3c" candidate_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99647dc7d1544e69ad5d3006624f890b" is_pre_election: true
I20260619 00:57:46.733552 19133 raft_consensus.cc:2781] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:46.733701 19133 raft_consensus.cc:482] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:46.733603 18795 raft_consensus.cc:2454] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82131fb9854b45e7ba2eecbf92d5ec18 in term 0.
I20260619 00:57:46.733822 19133 raft_consensus.cc:3037] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:46.736160 19133 raft_consensus.cc:504] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.736827 19133 leader_election.cc:288] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [CANDIDATE]: Term 1 election: Requested vote from peers 99647dc7d1544e69ad5d3006624f890b (127.0.97.1:35843), 0f2bb1d6fa1944d4ac6a0ab333766d5d (127.0.97.2:41445)
I20260619 00:57:46.737239 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9821c7d7f86d41d7b2aacd1730741d3c" candidate_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99647dc7d1544e69ad5d3006624f890b"
I20260619 00:57:46.737445 18795 raft_consensus.cc:3037] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:46.737414 18929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9821c7d7f86d41d7b2aacd1730741d3c" candidate_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d"
I20260619 00:57:46.737643 18929 raft_consensus.cc:3037] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:46.739670 18795 raft_consensus.cc:2454] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82131fb9854b45e7ba2eecbf92d5ec18 in term 1.
I20260619 00:57:46.739745 18929 raft_consensus.cc:2454] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82131fb9854b45e7ba2eecbf92d5ec18 in term 1.
I20260619 00:57:46.740097 18995 leader_election.cc:302] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82131fb9854b45e7ba2eecbf92d5ec18, 99647dc7d1544e69ad5d3006624f890b; no voters:
I20260619 00:57:46.740417 19133 raft_consensus.cc:2781] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:46.740967 19133 raft_consensus.cc:686] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 1 LEADER]: Becoming Leader. State: Replica: 82131fb9854b45e7ba2eecbf92d5ec18, State: Running, Role: LEADER
I20260619 00:57:46.741447 19133 consensus_queue.cc:237] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } }
I20260619 00:57:46.746183 18651 catalog_manager.cc:5495] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82131fb9854b45e7ba2eecbf92d5ec18 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82131fb9854b45e7ba2eecbf92d5ec18" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39425 } health_report { overall_health: HEALTHY } } }
I20260619 00:57:46.778656 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.772040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.97.3:39425)
0619 00:57:46.772409 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.772416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.772438 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:57:46.772538 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.772543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.772650 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.772837 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:46.773253 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.774217 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.776431 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.776994 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.777091 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.777566 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:46.778081 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:46.778210 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.778309 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.778376 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260619 00:57:46.787096 18929 raft_consensus.cc:1261] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 1 FOLLOWER]: Refusing update from remote peer 82131fb9854b45e7ba2eecbf92d5ec18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:46.787297 18795 raft_consensus.cc:1261] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 1 FOLLOWER]: Refusing update from remote peer 82131fb9854b45e7ba2eecbf92d5ec18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:46.788252 19133 consensus_queue.cc:1035] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99647dc7d1544e69ad5d3006624f890b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:46.788537 19136 consensus_queue.cc:1035] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2bb1d6fa1944d4ac6a0ab333766d5d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:46.801908 19140 mvcc.cc:205] Tried to move back new op lower bound from 7298378411148816384 to 7298378410976628736. Current Snapshot: MvccSnapshot[applied={T|T < 7298378411148816384}]
I20260619 00:57:46.807708 19142 mvcc.cc:205] Tried to move back new op lower bound from 7298378411148816384 to 7298378410976628736. Current Snapshot: MvccSnapshot[applied={T|T < 7298378411148816384}]
I20260619 00:57:46.808974 19139 mvcc.cc:205] Tried to move back new op lower bound from 7298378411148816384 to 7298378410976628736. Current Snapshot: MvccSnapshot[applied={T|T < 7298378411148816384 or (T in {7298378411148816384})}]
I20260619 00:57:46.819355 18651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:46.819679 18651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:46.823462 18651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6dadb13e16c4ad5831e49115e295859: Sending DeleteTablet for 3 replicas of tablet 9821c7d7f86d41d7b2aacd1730741d3c
I20260619 00:57:46.824647 18775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50174
I20260619 00:57:46.824786 18909 tablet_service.cc:1514] Processing DeleteTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57560
I20260619 00:57:46.824856 19042 tablet_service.cc:1514] Processing DeleteTablet for tablet 9821c7d7f86d41d7b2aacd1730741d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50830
I20260619 00:57:46.829746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18712
I20260619 00:57:46.836966 19149 tablet_replica.cc:330] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: stopping tablet replica
I20260619 00:57:46.837646 19149 raft_consensus.cc:2229] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:46.838441 19149 raft_consensus.cc:2258] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:46.839619 19150 tablet_replica.cc:330] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: stopping tablet replica
I20260619 00:57:46.840517 19151 tablet_replica.cc:330] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: stopping tablet replica
I20260619 00:57:46.841037 19151 raft_consensus.cc:2229] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:46.841445 19149 ts_tablet_manager.cc:1905] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:46.841718 19151 raft_consensus.cc:2258] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:46.842476 19150 raft_consensus.cc:2229] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:46.843205 19150 raft_consensus.cc:2258] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:46.844760 19151 ts_tablet_manager.cc:1905] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:46.845407 19150 ts_tablet_manager.cc:1905] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 00:57:46 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:57:46.853052 19150 ts_tablet_manager.cc:1918] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:46.853315 19150 log.cc:1196] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/9821c7d7f86d41d7b2aacd1730741d3c
I20260619 00:57:46.854017 19150 ts_tablet_manager.cc:1939] T 9821c7d7f86d41d7b2aacd1730741d3c P 82131fb9854b45e7ba2eecbf92d5ec18: Deleting consensus metadata
I20260619 00:57:46.856035 18636 catalog_manager.cc:4841] TS 82131fb9854b45e7ba2eecbf92d5ec18 (127.0.97.3:39425): tablet 9821c7d7f86d41d7b2aacd1730741d3c (table test-table [id=cdec70d7068b466da8f482b6d27566ee]) successfully deleted
I20260619 00:57:46.856611 19149 ts_tablet_manager.cc:1918] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:46.856931 19149 log.cc:1196] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/9821c7d7f86d41d7b2aacd1730741d3c
I20260619 00:57:46.857074 19151 ts_tablet_manager.cc:1918] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:46.857360 19151 log.cc:1196] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/9821c7d7f86d41d7b2aacd1730741d3c
I20260619 00:57:46.857668 19149 ts_tablet_manager.cc:1939] T 9821c7d7f86d41d7b2aacd1730741d3c P 99647dc7d1544e69ad5d3006624f890b: Deleting consensus metadata
I20260619 00:57:46.858096 19151 ts_tablet_manager.cc:1939] T 9821c7d7f86d41d7b2aacd1730741d3c P 0f2bb1d6fa1944d4ac6a0ab333766d5d: Deleting consensus metadata
I20260619 00:57:46.859709 18635 catalog_manager.cc:4841] TS 99647dc7d1544e69ad5d3006624f890b (127.0.97.1:35843): tablet 9821c7d7f86d41d7b2aacd1730741d3c (table test-table [id=cdec70d7068b466da8f482b6d27566ee]) successfully deleted
I20260619 00:57:46.859920 18636 catalog_manager.cc:4841] TS 0f2bb1d6fa1944d4ac6a0ab333766d5d (127.0.97.2:41445): tablet 9821c7d7f86d41d7b2aacd1730741d3c (table test-table [id=cdec70d7068b466da8f482b6d27566ee]) successfully deleted
I20260619 00:57:46.861281 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.831287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.0.97.1:35843)
0619 00:57:46.831464 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.831469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.839895 (+ 8426us) server_negotiation.cc:407] Connection header received
0619 00:57:46.840064 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.840070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.840177 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.840305 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:46.842305 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.843435 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.845405 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.846053 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.846171 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:46.853578 (+ 7407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:46.853592 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:46.853610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:46.853651 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:46.857613 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.858675 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.858685 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.858692 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.858870 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:46.859359 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:46.859365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:46.859368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:46.859880 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:46.860173 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:46.860731 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 00:57:46.860981 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:57:46.861884 18835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:46.965034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18712
I20260619 00:57:46.990808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18846
Jun 19 00:57:46 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:57:47.011063 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:46.991711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.97.2:41445)
0619 00:57:46.991885 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:46.991889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:46.991907 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:57:46.992080 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:46.992084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:46.992165 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:46.992254 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:46.994139 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.994989 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.997073 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:46.997477 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:46.997573 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:47.003133 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:47.003158 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:47.003182 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:47.003245 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:47.007705 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:47.008540 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:47.008551 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:47.008555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:47.008732 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:47.009209 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:47.009215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:47.009219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:47.009771 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:47.010127 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:47.010309 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 00:57:47.010570 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:57:47.011852 18969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:47.134192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18846
I20260619 00:57:47.157574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18979
Jun 19 00:57:47 dist-test-slave-r7bh krb5kdc[18606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:57:47.178059 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:47.158528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.97.3:39425)
0619 00:57:47.158707 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:47.158712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:47.158733 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:57:47.158961 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:47.158965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:47.159035 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:47.159138 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:47.161172 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:47.162038 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:47.163978 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:47.164438 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:47.164533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:47.170339 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:47.170358 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:47.170378 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:47.170429 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:47.174786 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:47.175530 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:47.175541 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:47.175545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:47.175744 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:47.176190 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:47.176196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:47.176201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:47.176709 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:47.177130 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:47.177301 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 00:57:47.177559 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:57:47.178889 19102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:47.303367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18979
I20260619 00:57:47.319772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 18619
I20260619 00:57:47.321076 18681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:47.435639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 18619
2026-06-19T00:57:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4492 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T00:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:47Z Disabled control of system clock
I20260619 00:57:47.512730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:47.847886 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:47.848181 19160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:47.848335 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:47.855654 19160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:47.855760 19160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:47.855835 19160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:47.855945 19160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:47.856036 19160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:47.865424 19160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:47.866791 19160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:47.868579 19160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:47.880511 19166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:47.880471 19165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:47.880686 19168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:47.881302 19160 server_base.cc:1034] running on GCE node
I20260619 00:57:47.882099 19160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:47.883541 19160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:47.884773 19160 hybrid_clock.cc:648] HybridClock initialized: now 1781830667884737 us; error 62 us; skew 500 ppm
I20260619 00:57:47.885180 19160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:47.892666 19160 webserver.cc:460] Webserver started at http://127.0.97.62:37357/ using document root <none> and password file <none>
I20260619 00:57:47.893513 19160 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:47.893664 19160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:47.894011 19160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:47.897269 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8"
format_stamp: "Formatted at 2026-06-19 00:57:47 on dist-test-slave-r7bh"
server_key: "88de5f030649ab207aaa314fe3cc065a"
server_key_iv: "34223c0895c57bd692442d44ede94073"
server_key_version: "encryptionkey@0"
I20260619 00:57:47.898072 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8"
format_stamp: "Formatted at 2026-06-19 00:57:47 on dist-test-slave-r7bh"
server_key: "88de5f030649ab207aaa314fe3cc065a"
server_key_iv: "34223c0895c57bd692442d44ede94073"
server_key_version: "encryptionkey@0"
I20260619 00:57:47.903278 19160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:57:47.906910 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:47.908385 19160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:57:47.908595 19160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8"
format_stamp: "Formatted at 2026-06-19 00:57:47 on dist-test-slave-r7bh"
server_key: "88de5f030649ab207aaa314fe3cc065a"
server_key_iv: "34223c0895c57bd692442d44ede94073"
server_key_version: "encryptionkey@0"
I20260619 00:57:47.908883 19160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:47.923970 19160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:47.924927 19160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:47.925230 19160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:47.943428 19160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45049
I20260619 00:57:47.943436 19225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45049 every 8 connection(s)
I20260619 00:57:47.944856 19160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:47.947053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19160
I20260619 00:57:47.947254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:47.947783 388 external_mini_cluster.cc:1428] Setting key a2f475292c63810a50801b65c9e62c70
I20260619 00:57:47.948988 19226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:47.963850 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:47.949855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.97.62:45049)
0619 00:57:47.950981 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:47.950998 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:47.951051 (+ 53us) server_negotiation.cc:407] Connection header received
0619 00:57:47.952834 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:47.952863 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:47.953493 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:47.953957 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:47.954512 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:47.956585 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:47.958085 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:47.959134 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:47.959330 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:47.960650 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:47.960662 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:47.960683 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:47.961507 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:47.961632 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:47.961665 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:47.961800 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:47.962047 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 00:57:47.962437 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":363,"threads_started":1}
I20260619 00:57:47.965126 19226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: Bootstrap starting.
I20260619 00:57:47.968689 19226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:47.969907 19226 log.cc:824] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:47.973029 19226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: No bootstrap required, opened a new log
I20260619 00:57:47.980543 19226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:57:47.981072 19226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:47.981201 19226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b46a5f3d3b465ab813d1d6ce05a2b8, State: Initialized, Role: FOLLOWER
I20260619 00:57:47.981884 19226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:57:47.982128 19226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:47.982263 19226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:47.982450 19226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:47.984759 19226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:57:47.985536 19226 leader_election.cc:302] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b46a5f3d3b465ab813d1d6ce05a2b8; no voters:
I20260619 00:57:47.986236 19226 leader_election.cc:288] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:47.986400 19231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:47.987150 19231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 22b46a5f3d3b465ab813d1d6ce05a2b8, State: Running, Role: LEADER
I20260619 00:57:47.987748 19231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } }
I20260619 00:57:47.988564 19226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:47.991048 19232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260619 00:57:47.990960 19233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b46a5f3d3b465ab813d1d6ce05a2b8. Latest consensus state: current_term: 1 leader_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b46a5f3d3b465ab813d1d6ce05a2b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45049 } } }
I20260619 00:57:47.991482 19232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:47.991510 19233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:47.993602 19239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:48.001312 19239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:48.010995 19239 catalog_manager.cc:1347] Generated new cluster ID: 2ba28ef52a174f2e8e0e91b7a6281f9a
I20260619 00:57:48.011121 19239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:48.050338 19239 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:48.051628 19239 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:48.059468 19239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: Generated new TSK 0
I20260619 00:57:48.060566 19239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:57:48.072938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:48.411124 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:48.411427 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:48.411598 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:48.419056 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:48.419178 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:48.419363 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:48.428627 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:48.429973 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:48.431747 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:48.445142 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:48.445217 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:48.445195 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:48.445917 19250 server_base.cc:1034] running on GCE node
I20260619 00:57:48.446601 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:48.447645 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:48.448977 19250 hybrid_clock.cc:648] HybridClock initialized: now 1781830668448920 us; error 45 us; skew 500 ppm
I20260619 00:57:48.449331 19250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:48.451691 19250 webserver.cc:460] Webserver started at http://127.0.97.1:45113/ using document root <none> and password file <none>
I20260619 00:57:48.452418 19250 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:48.452554 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:48.452960 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:48.456202 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "0ba24fd7884f4a62a056d123466f9e17"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "deefd6d93ad35c96b40d423e94dc56d5"
server_key_iv: "3de05438fca5553fe1143aef148e109f"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.457134 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba24fd7884f4a62a056d123466f9e17"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "deefd6d93ad35c96b40d423e94dc56d5"
server_key_iv: "3de05438fca5553fe1143aef148e109f"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.463477 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 00:57:48.467374 19264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.468791 19250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:57:48.469041 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "0ba24fd7884f4a62a056d123466f9e17"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "deefd6d93ad35c96b40d423e94dc56d5"
server_key_iv: "3de05438fca5553fe1143aef148e109f"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.469285 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:48.481642 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:48.482450 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:48.482730 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:48.483644 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:48.485188 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:48.485320 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.485450 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:48.485543 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.519249 19250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41155
I20260619 00:57:48.519344 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41155 every 8 connection(s)
I20260619 00:57:48.520605 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:48.529170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19250
I20260619 00:57:48.529364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:48.529946 388 external_mini_cluster.cc:1428] Setting key f4c5fcf310f976bc9e276814bef67cff
I20260619 00:57:48.533089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:48.534857 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:48.523339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54905 (local address 127.0.97.62:45049)
0619 00:57:48.523819 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:48.523826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:48.525263 (+ 1437us) server_negotiation.cc:407] Connection header received
0619 00:57:48.527143 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:48.527147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:48.527230 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:48.527335 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:48.528631 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:48.529554 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:48.531734 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:48.532238 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:48.532338 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:48.534075 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:48.534080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:48.534083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:48.534102 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:48.534148 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:48.534161 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:48.534232 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:48.534522 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 00:57:48.534604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260619 00:57:48.536459 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:48.524046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.1:54905)
0619 00:57:48.525095 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:48.525141 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:48.526814 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:48.527730 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:48.527740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:48.527839 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:48.528447 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:48.528471 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:48.529730 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:48.529735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:48.531541 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:48.531556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:48.532482 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:48.532490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:48.532857 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:48.533672 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:48.533679 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:48.533740 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:48.533856 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:48.534265 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:48.534279 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:48.534291 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 00:57:48.534499 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 00:57:48.534979 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":301,"threads_started":1}
I20260619 00:57:48.538144 19377 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:57:48.538509 19377 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:48.539278 19377 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:57:48.541615 19191 ts_manager.cc:194] Registered new tserver with Master: 0ba24fd7884f4a62a056d123466f9e17 (127.0.97.1:41155)
I20260619 00:57:48.543156 19191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54905
W20260619 00:57:48.865134 19381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:48.865434 19381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:48.865576 19381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:48.873081 19381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:48.873184 19381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:48.873344 19381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:48.882805 19381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:48.884097 19381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:48.885957 19381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:48.900101 19387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:48.900177 19389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:48.900169 19386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:48.901142 19381 server_base.cc:1034] running on GCE node
I20260619 00:57:48.901823 19381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:48.902858 19381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:48.904100 19381 hybrid_clock.cc:648] HybridClock initialized: now 1781830668904046 us; error 64 us; skew 500 ppm
I20260619 00:57:48.904456 19381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:48.907858 19381 webserver.cc:460] Webserver started at http://127.0.97.2:46285/ using document root <none> and password file <none>
I20260619 00:57:48.908581 19381 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:48.908672 19381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:48.909037 19381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:48.912181 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "9d6ac679f0cf4138a083cbd7e002fec4"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "f383311f8c58abe0277cc0659f6de374"
server_key_iv: "81a1a2cc0ca31ea81fba767b370e55ea"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.913098 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9d6ac679f0cf4138a083cbd7e002fec4"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "f383311f8c58abe0277cc0659f6de374"
server_key_iv: "81a1a2cc0ca31ea81fba767b370e55ea"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.918280 19381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:57:48.921725 19395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.923239 19381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:48.923413 19381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "9d6ac679f0cf4138a083cbd7e002fec4"
format_stamp: "Formatted at 2026-06-19 00:57:48 on dist-test-slave-r7bh"
server_key: "f383311f8c58abe0277cc0659f6de374"
server_key_iv: "81a1a2cc0ca31ea81fba767b370e55ea"
server_key_version: "encryptionkey@0"
I20260619 00:57:48.923641 19381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:48.950814 19381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:48.951663 19381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:48.952000 19381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:48.953075 19381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:48.954582 19381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:48.954666 19381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.954815 19381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:48.954922 19381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:48.990898 19381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36545
I20260619 00:57:48.990927 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36545 every 8 connection(s)
I20260619 00:57:48.992331 19381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:49.001164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19381
I20260619 00:57:49.001379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:49.001919 388 external_mini_cluster.cc:1428] Setting key d9a91b35a67281ca0d56ea4fb547c95e
I20260619 00:57:49.005317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--builtin_ntp_servers=127.0.97.20:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:49.006558 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:48.995006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48383 (local address 127.0.97.62:45049)
0619 00:57:48.995216 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:48.995224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:48.996783 (+ 1559us) server_negotiation.cc:407] Connection header received
0619 00:57:48.998755 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:48.998759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:48.998849 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:48.998961 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:49.000128 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.000975 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.003213 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.003857 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.004028 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.005782 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:49.005787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:49.005790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:49.005810 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:49.005843 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:49.005855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:49.005937 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.006224 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.006310 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:57:49.008267 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:48.995636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.2:48383)
0619 00:57:48.996588 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:48.996651 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:48.998467 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:48.999169 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:48.999178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:48.999275 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:48.999950 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:48.999968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.001128 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.001136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.003073 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.003085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.004189 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.004199 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.004525 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.005352 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:49.005357 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:49.005443 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:49.005575 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:49.005970 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:49.005983 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:49.005996 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.006213 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.006708 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":269,"threads_started":1}
I20260619 00:57:49.010085 19508 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:57:49.010455 19508 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:49.011533 19508 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:57:49.013396 19191 ts_manager.cc:194] Registered new tserver with Master: 9d6ac679f0cf4138a083cbd7e002fec4 (127.0.97.2:36545)
I20260619 00:57:49.014343 19191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48383
W20260619 00:57:49.353664 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:49.354089 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:49.354260 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:49.362142 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:49.362248 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:49.362366 19511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:49.371868 19511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:49.373373 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:49.375196 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:49.388649 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:49.388705 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:49.388674 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:49.389518 19511 server_base.cc:1034] running on GCE node
I20260619 00:57:49.390199 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:49.391158 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:49.392403 19511 hybrid_clock.cc:648] HybridClock initialized: now 1781830669392369 us; error 65 us; skew 500 ppm
I20260619 00:57:49.392733 19511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:49.394984 19511 webserver.cc:460] Webserver started at http://127.0.97.3:40835/ using document root <none> and password file <none>
I20260619 00:57:49.395712 19511 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:49.395836 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:49.396190 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:49.399156 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3"
format_stamp: "Formatted at 2026-06-19 00:57:49 on dist-test-slave-r7bh"
server_key: "35836001419c6d1a3f12902b41220aa1"
server_key_iv: "76aec827da62e7090c9a3f1754990af0"
server_key_version: "encryptionkey@0"
I20260619 00:57:49.399940 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3"
format_stamp: "Formatted at 2026-06-19 00:57:49 on dist-test-slave-r7bh"
server_key: "35836001419c6d1a3f12902b41220aa1"
server_key_iv: "76aec827da62e7090c9a3f1754990af0"
server_key_version: "encryptionkey@0"
I20260619 00:57:49.405048 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:57:49.408520 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:49.409951 19511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:49.410130 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3"
format_stamp: "Formatted at 2026-06-19 00:57:49 on dist-test-slave-r7bh"
server_key: "35836001419c6d1a3f12902b41220aa1"
server_key_iv: "76aec827da62e7090c9a3f1754990af0"
server_key_version: "encryptionkey@0"
I20260619 00:57:49.410368 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:49.447798 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:49.448716 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:49.449041 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:49.450088 19511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:49.451630 19511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:49.451753 19511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:49.451890 19511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:49.451980 19511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:49.485441 19511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45715
I20260619 00:57:49.485486 19637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45715 every 8 connection(s)
I20260619 00:57:49.486842 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:49.494315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19511
I20260619 00:57:49.494525 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:49.494972 388 external_mini_cluster.cc:1428] Setting key 1fa94a2b6bb647301538ba016b08208b
I20260619 00:57:49.500339 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.489600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36759 (local address 127.0.97.62:45049)
0619 00:57:49.489790 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.489794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.491243 (+ 1449us) server_negotiation.cc:407] Connection header received
0619 00:57:49.493022 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.493027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.493108 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.493201 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:49.494450 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.495447 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.497590 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.498105 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.498200 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.499632 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:49.499638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:49.499642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:49.499662 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:49.499698 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:49.499712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:49.499809 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.499990 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.500073 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:57:49.501395 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.490189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45049 (local address 127.0.97.3:36759)
0619 00:57:49.491076 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.491118 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.492697 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.493397 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.493404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.493501 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:49.494209 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.494234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.495601 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.495605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.497424 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.497440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.498300 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.498305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.498503 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.499299 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:49.499301 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:49.499340 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:49.499421 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:49.499807 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:49.499814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:49.499820 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.499958 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.500265 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":256,"threads_started":1}
I20260619 00:57:49.502796 19638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45049
I20260619 00:57:49.503096 19638 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:49.503813 19638 heartbeater.cc:511] Master 127.0.97.62:45049 requested a full tablet report, sending...
I20260619 00:57:49.505347 19191 ts_manager.cc:194] Registered new tserver with Master: a50662ac4d9a4e20b2b66ef7b662b0a3 (127.0.97.3:45715)
I20260619 00:57:49.506135 19191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36759
I20260619 00:57:49.510341 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:49.520955 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.514599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.97.62:45049)
0619 00:57:49.514811 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.514816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.515224 (+ 408us) server_negotiation.cc:407] Connection header received
0619 00:57:49.515356 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.515359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.515429 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.515511 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:49.516031 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.516766 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.518330 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.518952 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.519078 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.520174 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:49.520180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:49.520185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:49.520206 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:49.520242 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:49.520257 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:49.520391 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.520557 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.520616 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:57:49.525609 19191 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:49.532990 19191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:49.546955 19377 heartbeater.cc:503] Master 127.0.97.62:45049 was elected leader, sending a full tablet report...
I20260619 00:57:49.559844 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.545959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36545 (local address 127.0.0.1:45622)
0619 00:57:49.547378 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.547422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.547594 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.550195 (+ 2601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.550206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.550252 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:49.550844 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.550861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.553434 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.553440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.556605 (+ 3165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.556632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.558812 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.558820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.559080 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.559157 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.559275 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.559490 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":2117,"threads_started":1}
I20260619 00:57:49.560616 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.545987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.0.97.2:36545)
0619 00:57:49.549462 (+ 3475us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.549472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.549508 (+ 36us) server_negotiation.cc:407] Connection header received
0619 00:57:49.549616 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.549624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.549919 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.551144 (+ 1225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:49.551466 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.553270 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.557018 (+ 3748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.558571 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.559656 (+ 1085us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.559934 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.560015 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.560215 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2659,"thread_start_us":2545,"threads_started":1}
I20260619 00:57:49.564700 19443 tablet_service.cc:1467] Processing CreateTablet for tablet 9724c35b959a4c669509bf6476993565 (DEFAULT_TABLE table=test-table [id=b9096b20f1404723ae381ea56e2cc7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:49.566310 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9724c35b959a4c669509bf6476993565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:49.566970 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.547944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43686)
0619 00:57:49.558898 (+ 10954us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.558920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.559059 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.560066 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.560070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.560100 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:49.561020 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.561032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.562888 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.562893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.564714 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.564727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.566198 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.566204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.566363 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.566397 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.566477 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.566609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10817,"mutex_wait_us":130,"thread_start_us":10650,"threads_started":1}
I20260619 00:57:49.567126 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.548941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.0.97.1:41155)
0619 00:57:49.557471 (+ 8530us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.557484 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.559048 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 00:57:49.559230 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.559242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.559573 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.559790 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:49.561210 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.562720 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.564941 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.566001 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.566164 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.566447 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.566546 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.566726 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8273,"thread_start_us":9266,"threads_started":1}
I20260619 00:57:49.567935 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.545963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45715 (local address 127.0.0.1:34912)
0619 00:57:49.548189 (+ 2226us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.548209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.548312 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.560279 (+ 11967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.560285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.560316 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:49.560844 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.560862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.563053 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.563060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.565323 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.565339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.567180 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.567186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.567384 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.567429 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.567490 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.567612 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"mutex_wait_us":239,"thread_start_us":777,"threads_started":1}
I20260619 00:57:49.569314 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.546526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.0.97.3:45715)
0619 00:57:49.559642 (+ 13116us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.559652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.559686 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:57:49.559780 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.559788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.560044 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.560646 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:49.561020 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.562893 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.565561 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.566960 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.568194 (+ 1234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.568703 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.568773 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.568962 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12925,"thread_start_us":12762,"threads_started":1}
I20260619 00:57:49.570837 19312 tablet_service.cc:1467] Processing CreateTablet for tablet 9724c35b959a4c669509bf6476993565 (DEFAULT_TABLE table=test-table [id=b9096b20f1404723ae381ea56e2cc7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:49.572324 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9724c35b959a4c669509bf6476993565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:49.572961 19573 tablet_service.cc:1467] Processing CreateTablet for tablet 9724c35b959a4c669509bf6476993565 (DEFAULT_TABLE table=test-table [id=b9096b20f1404723ae381ea56e2cc7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:49.574455 19573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9724c35b959a4c669509bf6476993565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:49.579398 19656 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Bootstrap starting.
I20260619 00:57:49.582955 19656 tablet_bootstrap.cc:654] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:49.584069 19656 log.cc:824] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:49.589007 19658 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Bootstrap starting.
I20260619 00:57:49.589232 19656 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: No bootstrap required, opened a new log
I20260619 00:57:49.589533 19656 ts_tablet_manager.cc:1397] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260619 00:57:49.590811 19659 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Bootstrap starting.
I20260619 00:57:49.593111 19658 tablet_bootstrap.cc:654] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:49.593974 19659 tablet_bootstrap.cc:654] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:49.594408 19658 log.cc:824] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:49.595045 19659 log.cc:824] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:49.597836 19658 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: No bootstrap required, opened a new log
I20260619 00:57:49.598143 19658 ts_tablet_manager.cc:1397] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 00:57:49.598279 19659 tablet_bootstrap.cc:492] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: No bootstrap required, opened a new log
I20260619 00:57:49.598558 19659 ts_tablet_manager.cc:1397] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 00:57:49.600051 19656 raft_consensus.cc:348] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.600443 19656 raft_consensus.cc:374] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:49.600543 19656 raft_consensus.cc:729] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6ac679f0cf4138a083cbd7e002fec4, State: Initialized, Role: FOLLOWER
I20260619 00:57:49.601181 19656 consensus_queue.cc:260] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.602289 19508 heartbeater.cc:503] Master 127.0.97.62:45049 was elected leader, sending a full tablet report...
I20260619 00:57:49.604221 19656 ts_tablet_manager.cc:1428] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260619 00:57:49.609483 19659 raft_consensus.cc:348] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.609483 19658 raft_consensus.cc:348] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.610019 19659 raft_consensus.cc:374] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:49.610028 19658 raft_consensus.cc:374] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:49.610136 19658 raft_consensus.cc:729] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba24fd7884f4a62a056d123466f9e17, State: Initialized, Role: FOLLOWER
I20260619 00:57:49.610136 19659 raft_consensus.cc:729] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50662ac4d9a4e20b2b66ef7b662b0a3, State: Initialized, Role: FOLLOWER
I20260619 00:57:49.610836 19659 consensus_queue.cc:260] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.610836 19658 consensus_queue.cc:260] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.612169 19638 heartbeater.cc:503] Master 127.0.97.62:45049 was elected leader, sending a full tablet report...
I20260619 00:57:49.613128 19659 ts_tablet_manager.cc:1428] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 00:57:49.613418 19658 ts_tablet_manager.cc:1428] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260619 00:57:49.738726 19639 tablet.cc:2369] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:49.744122 19662 raft_consensus.cc:482] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260619 00:57:49.744560 19509 tablet.cc:2369] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:49.744568 19662 raft_consensus.cc:504] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.746508 19662 leader_election.cc:288] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a50662ac4d9a4e20b2b66ef7b662b0a3 (127.0.97.3:45715), 0ba24fd7884f4a62a056d123466f9e17 (127.0.97.1:41155)
I20260619 00:57:49.754798 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.747048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53495 (local address 127.0.97.3:45715)
0619 00:57:49.747249 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.747253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.747594 (+ 341us) server_negotiation.cc:407] Connection header received
0619 00:57:49.747799 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.747806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.747892 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.748027 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:49.748838 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.749860 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.752445 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.753716 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.753861 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.753971 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.754418 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.754544 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:57:49.754798 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.747896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59265 (local address 127.0.97.1:41155)
0619 00:57:49.748115 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.748120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.748316 (+ 196us) server_negotiation.cc:407] Connection header received
0619 00:57:49.748518 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.748522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.748595 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.748670 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:49.749280 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.750207 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.752577 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.753807 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.753917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.754009 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.754402 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.754553 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 00:57:49.754983 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.746792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45715 (local address 127.0.97.2:53495)
0619 00:57:49.747444 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.747471 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.747625 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.748144 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.748150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.748175 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:49.748635 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.748667 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.749997 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.750001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.752218 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.752233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.753935 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.753942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.754221 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.754266 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.754357 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.754515 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":72,"spinlock_wait_cycles":25856,"thread_start_us":335,"threads_started":1}
I20260619 00:57:49.755404 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.747582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.97.2:59265)
0619 00:57:49.748209 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.748232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:49.748367 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:49.748721 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:49.748728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:49.748755 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:49.749127 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.749135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.750342 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.750348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.752410 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:49.752423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.753981 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.753986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:49.754207 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.754261 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 00:57:49.754357 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 00:57:49.754963 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":430,"thread_start_us":344,"threads_started":1}
I20260619 00:57:49.756244 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9724c35b959a4c669509bf6476993565" candidate_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" is_pre_election: true
I20260619 00:57:49.756379 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9724c35b959a4c669509bf6476993565" candidate_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba24fd7884f4a62a056d123466f9e17" is_pre_election: true
I20260619 00:57:49.756783 19593 raft_consensus.cc:2454] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6ac679f0cf4138a083cbd7e002fec4 in term 0.
I20260619 00:57:49.756974 19332 raft_consensus.cc:2454] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6ac679f0cf4138a083cbd7e002fec4 in term 0.
I20260619 00:57:49.757606 19396 leader_election.cc:302] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6ac679f0cf4138a083cbd7e002fec4, a50662ac4d9a4e20b2b66ef7b662b0a3; no voters:
I20260619 00:57:49.757977 19662 raft_consensus.cc:2781] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:49.758126 19662 raft_consensus.cc:482] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:49.758251 19662 raft_consensus.cc:3037] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:49.760450 19662 raft_consensus.cc:504] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.761165 19662 leader_election.cc:288] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [CANDIDATE]: Term 1 election: Requested vote from peers a50662ac4d9a4e20b2b66ef7b662b0a3 (127.0.97.3:45715), 0ba24fd7884f4a62a056d123466f9e17 (127.0.97.1:41155)
I20260619 00:57:49.761579 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9724c35b959a4c669509bf6476993565" candidate_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3"
I20260619 00:57:49.761631 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9724c35b959a4c669509bf6476993565" candidate_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba24fd7884f4a62a056d123466f9e17"
I20260619 00:57:49.761782 19593 raft_consensus.cc:3037] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:49.761824 19332 raft_consensus.cc:3037] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:49.764003 19593 raft_consensus.cc:2454] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6ac679f0cf4138a083cbd7e002fec4 in term 1.
I20260619 00:57:49.764014 19332 raft_consensus.cc:2454] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6ac679f0cf4138a083cbd7e002fec4 in term 1.
I20260619 00:57:49.764428 19396 leader_election.cc:302] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6ac679f0cf4138a083cbd7e002fec4, a50662ac4d9a4e20b2b66ef7b662b0a3; no voters:
I20260619 00:57:49.764770 19662 raft_consensus.cc:2781] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:49.765278 19662 raft_consensus.cc:686] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6ac679f0cf4138a083cbd7e002fec4, State: Running, Role: LEADER
I20260619 00:57:49.765868 19662 consensus_queue.cc:237] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } }
I20260619 00:57:49.770794 19191 catalog_manager.cc:5495] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d6ac679f0cf4138a083cbd7e002fec4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6ac679f0cf4138a083cbd7e002fec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 } health_report { overall_health: UNKNOWN } } }
W20260619 00:57:49.775354 19378 tablet.cc:2369] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:49.800405 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.794581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.0.97.2:36545)
0619 00:57:49.794786 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.794794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.794848 (+ 54us) server_negotiation.cc:407] Connection header received
0619 00:57:49.795074 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.795080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.795192 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.795358 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:49.795759 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.796691 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.798076 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.798500 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.798644 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.799225 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:49.799788 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:49.799979 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.800100 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.800143 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:57:49.809192 19593 raft_consensus.cc:1261] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 1 FOLLOWER]: Refusing update from remote peer 9d6ac679f0cf4138a083cbd7e002fec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:49.809192 19332 raft_consensus.cc:1261] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 1 FOLLOWER]: Refusing update from remote peer 9d6ac679f0cf4138a083cbd7e002fec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:49.810173 19662 consensus_queue.cc:1035] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba24fd7884f4a62a056d123466f9e17" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:49.810491 19667 consensus_queue.cc:1035] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a50662ac4d9a4e20b2b66ef7b662b0a3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:49.823704 19672 mvcc.cc:205] Tried to move back new op lower bound from 7298378423527124992 to 7298378423364898816. Current Snapshot: MvccSnapshot[applied={T|T < 7298378423527124992}]
I20260619 00:57:49.837030 19190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:49.837378 19190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:49.841199 19190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22b46a5f3d3b465ab813d1d6ce05a2b8: Sending DeleteTablet for 3 replicas of tablet 9724c35b959a4c669509bf6476993565
I20260619 00:57:49.842360 19443 tablet_service.cc:1514] Processing DeleteTablet for tablet 9724c35b959a4c669509bf6476993565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:49 UTC) from {username='slave'} at 127.0.0.1:45622
I20260619 00:57:49.842626 19312 tablet_service.cc:1514] Processing DeleteTablet for tablet 9724c35b959a4c669509bf6476993565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:49 UTC) from {username='slave'} at 127.0.0.1:43686
I20260619 00:57:49.843246 19573 tablet_service.cc:1514] Processing DeleteTablet for tablet 9724c35b959a4c669509bf6476993565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:49 UTC) from {username='slave'} at 127.0.0.1:34912
I20260619 00:57:49.844911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19250
I20260619 00:57:49.856256 19679 tablet_replica.cc:330] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: stopping tablet replica
I20260619 00:57:49.856446 19680 tablet_replica.cc:330] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: stopping tablet replica
I20260619 00:57:49.856772 19679 raft_consensus.cc:2229] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:49.857074 19680 raft_consensus.cc:2229] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:49.857514 19679 raft_consensus.cc:2258] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:49.857720 19680 raft_consensus.cc:2258] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:49.858328 19681 tablet_replica.cc:330] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: stopping tablet replica
I20260619 00:57:49.858835 19681 raft_consensus.cc:2229] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:49.859395 19681 raft_consensus.cc:2258] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:49.859849 19679 ts_tablet_manager.cc:1905] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:49.860908 19680 ts_tablet_manager.cc:1905] T 9724c35b959a4c669509bf6476993565 P 0ba24fd7884f4a62a056d123466f9e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:49.862669 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.846624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.97.1:41155)
0619 00:57:49.846846 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.846855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.851174 (+ 4319us) server_negotiation.cc:407] Connection header received
0619 00:57:49.852277 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.852286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.852429 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.852535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:49.853968 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.855488 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.858206 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.858824 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.858938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.860959 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:49.860970 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:49.860988 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:49.861744 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:49.861820 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:49.861845 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:49.861944 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.862189 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.862305 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:57:49.864164 19372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:49.865783 19681 ts_tablet_manager.cc:1905] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:49.868366 19679 ts_tablet_manager.cc:1918] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:49.868592 19679 log.cc:1196] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/9724c35b959a4c669509bf6476993565
I20260619 00:57:49.869494 19679 ts_tablet_manager.cc:1939] T 9724c35b959a4c669509bf6476993565 P 9d6ac679f0cf4138a083cbd7e002fec4: Deleting consensus metadata
I20260619 00:57:49.871533 19681 ts_tablet_manager.cc:1918] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:49.871783 19681 log.cc:1196] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/9724c35b959a4c669509bf6476993565
I20260619 00:57:49.872126 19178 catalog_manager.cc:4841] TS 9d6ac679f0cf4138a083cbd7e002fec4 (127.0.97.2:36545): tablet 9724c35b959a4c669509bf6476993565 (table test-table [id=b9096b20f1404723ae381ea56e2cc7d6]) successfully deleted
I20260619 00:57:49.872368 19681 ts_tablet_manager.cc:1939] T 9724c35b959a4c669509bf6476993565 P a50662ac4d9a4e20b2b66ef7b662b0a3: Deleting consensus metadata
I20260619 00:57:49.874349 19175 catalog_manager.cc:4841] TS a50662ac4d9a4e20b2b66ef7b662b0a3 (127.0.97.3:45715): tablet 9724c35b959a4c669509bf6476993565 (table test-table [id=b9096b20f1404723ae381ea56e2cc7d6]) successfully deleted
I20260619 00:57:49.968019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19250
W20260619 00:57:49.987371 19175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41155 (error 108)
I20260619 00:57:49.987519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19381
I20260619 00:57:49.989547 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.988992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43702)
0619 00:57:49.989203 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:49.989374 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 00:57:49.989900 19175 catalog_manager.cc:4584] TS 0ba24fd7884f4a62a056d123466f9e17 (127.0.97.1:41155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9724c35b959a4c669509bf6476993565: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
I20260619 00:57:49.996605 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:49.988674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.97.2:36545)
0619 00:57:49.988880 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:49.988885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:49.988908 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:57:49.988970 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:49.988974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:49.989081 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:49.989185 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:49.989734 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.990973 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.992800 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:49.993510 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:49.993635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:49.994868 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:49.994879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:49.994894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:49.995710 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:49.995790 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:49.995821 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:49.995928 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:49.996104 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 00:57:49.996214 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:57:49.997380 19503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:50.043844 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.043245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43716)
0619 00:57:50.043490 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.043606 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 00:57:50.045173 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.044717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43718)
0619 00:57:50.044934 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.045023 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 00:57:50.105778 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.105252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43720)
0619 00:57:50.105489 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.105610 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 00:57:50.106840 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.106448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43722)
0619 00:57:50.106582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.106681 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 00:57:50.108780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19381
I20260619 00:57:50.127285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19511
I20260619 00:57:50.134042 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.128014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.0.97.3:45715)
0619 00:57:50.128192 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:50.128200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:50.128223 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:57:50.128440 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:50.128445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:50.128530 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:50.128655 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:50.129112 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:50.129821 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:50.131128 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:50.131518 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:50.131650 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:50.132679 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:50.132688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:50.132702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:50.133318 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:50.133364 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:50.133383 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:50.133518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:50.133633 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 00:57:50.133735 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:57:50.134655 19633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:50.188205 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.187626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43726)
0619 00:57:50.187876 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.187995 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 00:57:50.189430 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:50.188982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41155 (local address 127.0.0.1:43730)
0619 00:57:50.189199 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:50.189293 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 00:57:50.225979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19511
I20260619 00:57:50.242441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19160
I20260619 00:57:50.243497 19221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:50.354861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19160
2026-06-19T00:57:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2920 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:57:50 dist-test-slave-r7bh krb5kdc[19690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:57:50 dist-test-slave-r7bh krb5kdc[19690](info): set up 2 sockets
Jun 19 00:57:50 dist-test-slave-r7bh krb5kdc[19690](info): commencing operation
krb5kdc: starting...
W20260619 00:57:51.473244 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:57:51 dist-test-slave-r7bh krb5kdc[19690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:57:51.725742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40045
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:52.064503 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:52.064908 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:52.065074 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:52.072373 19703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:52.072540 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:52.072659 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:52.072765 19703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:52.072878 19703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:52.083508 19703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:52.085021 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:52.086377 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:52.098208 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:52.098174 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:52.098284 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:52.099386 19703 server_base.cc:1034] running on GCE node
I20260619 00:57:52.100323 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:52.101979 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:52.103233 19703 hybrid_clock.cc:648] HybridClock initialized: now 1781830672103187 us; error 64 us; skew 500 ppm
I20260619 00:57:52.103600 19703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:52 dist-test-slave-r7bh krb5kdc[19690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:52.106892 19703 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:57:52.108342 19703 webserver.cc:460] Webserver started at http://127.0.97.62:45989/ using document root <none> and password file <none>
I20260619 00:57:52.109122 19703 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:52.109225 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:52.109484 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:52.112447 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "de9683b22a3f4a67863dc6157e306335"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "f20d0085a80132e70abc9091b932b58d"
server_key_iv: "fecfe96b10c4edcadd0745f7feb06fa5"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.113217 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "de9683b22a3f4a67863dc6157e306335"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "f20d0085a80132e70abc9091b932b58d"
server_key_iv: "fecfe96b10c4edcadd0745f7feb06fa5"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.118213 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:57:52.121704 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:52.123066 19703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:52.123214 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "de9683b22a3f4a67863dc6157e306335"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "f20d0085a80132e70abc9091b932b58d"
server_key_iv: "fecfe96b10c4edcadd0745f7feb06fa5"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.123386 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:52.148340 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:52.153126 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:52.153370 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:52.172420 19703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40045
I20260619 00:57:52.172447 19769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40045 every 8 connection(s)
I20260619 00:57:52.173939 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:52.177955 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:52.181565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19703
I20260619 00:57:52.181759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:52.182303 388 external_mini_cluster.cc:1428] Setting key d8272aaf822b18cd2096babb93189fa7
I20260619 00:57:52.191144 19770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: Bootstrap starting.
I20260619 00:57:52.195178 19770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:52.196578 19770 log.cc:824] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:57:52 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:52.200230 19770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: No bootstrap required, opened a new log
I20260619 00:57:52.207860 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:52.184313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.0.97.62:40045)
0619 00:57:52.185142 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:52.185155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:52.185198 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:57:52.186870 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:52.186894 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:52.187346 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:52.187696 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:52.189626 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.190963 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.192855 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.193562 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.193697 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:52.198926 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:52.198938 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:52.198964 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:52.199008 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:52.203094 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:52.203843 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:52.203854 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:52.203863 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:52.204093 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:52.204481 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:52.204486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:52.204490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:52.205231 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:52.205546 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:52.205897 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 00:57:52.206395 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":403,"threads_started":1}
I20260619 00:57:52.211051 19770 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } }
I20260619 00:57:52.211459 19770 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:52.211539 19770 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9683b22a3f4a67863dc6157e306335, State: Initialized, Role: FOLLOWER
I20260619 00:57:52.212116 19770 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } }
I20260619 00:57:52.212344 19770 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:52.212443 19770 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:52.212575 19770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:52.214807 19770 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } }
I20260619 00:57:52.215327 19770 leader_election.cc:302] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9683b22a3f4a67863dc6157e306335; no voters:
I20260619 00:57:52.216006 19770 leader_election.cc:288] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:52.216130 19775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:52.217393 19775 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [term 1 LEADER]: Becoming Leader. State: Replica: de9683b22a3f4a67863dc6157e306335, State: Running, Role: LEADER
I20260619 00:57:52.217939 19775 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } }
I20260619 00:57:52.218351 19770 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:52.220956 19777 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9683b22a3f4a67863dc6157e306335. Latest consensus state: current_term: 1 leader_uuid: "de9683b22a3f4a67863dc6157e306335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } } }
I20260619 00:57:52.221117 19776 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9683b22a3f4a67863dc6157e306335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9683b22a3f4a67863dc6157e306335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40045 } } }
I20260619 00:57:52.221482 19777 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:52.221479 19776 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:52.224253 19781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:52.229571 19781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:52.238673 19781 catalog_manager.cc:1347] Generated new cluster ID: 1cac9f896402497a8337b08a4cfc9b24
I20260619 00:57:52.238811 19781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:52.251675 19781 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:52.253111 19781 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:52.262306 19781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: Generated new TSK 0
I20260619 00:57:52.263204 19781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:57:52.328550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:52.667640 19796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:52.667946 19796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:52.668046 19796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:52.675515 19796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:52.675642 19796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:52.675765 19796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:52.685974 19796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:52.687263 19796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:52.688557 19796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:52.702107 19801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:52.702343 19802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:52.702145 19804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:52.703163 19796 server_base.cc:1034] running on GCE node
I20260619 00:57:52.703787 19796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:52.704799 19796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:52.706177 19796 hybrid_clock.cc:648] HybridClock initialized: now 1781830672706069 us; error 61 us; skew 500 ppm
I20260619 00:57:52.706568 19796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:52 dist-test-slave-r7bh krb5kdc[19690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:52.710639 19796 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:57:52.712239 19796 webserver.cc:460] Webserver started at http://127.0.97.1:34231/ using document root <none> and password file <none>
I20260619 00:57:52.713091 19796 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:52.713266 19796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:52.713616 19796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:52.716773 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "315819b7b4d547e6888ba166e384b522"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "ee6f79858ee392966b7a108875657992"
server_key_iv: "3d706e99a6a66f5b8d9f070468df030d"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.717715 19796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "315819b7b4d547e6888ba166e384b522"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "ee6f79858ee392966b7a108875657992"
server_key_iv: "3d706e99a6a66f5b8d9f070468df030d"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.723752 19796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 00:57:52.727593 19811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:52.729365 19796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:52.729563 19796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "315819b7b4d547e6888ba166e384b522"
format_stamp: "Formatted at 2026-06-19 00:57:52 on dist-test-slave-r7bh"
server_key: "ee6f79858ee392966b7a108875657992"
server_key_iv: "3d706e99a6a66f5b8d9f070468df030d"
server_key_version: "encryptionkey@0"
I20260619 00:57:52.729794 19796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:52.783411 19796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:52.788277 19796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:52.788590 19796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:52.789672 19796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:52.791042 19796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:52.791155 19796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:52.791308 19796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:52.791384 19796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:52.825730 19796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44471
I20260619 00:57:52.825793 19923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44471 every 8 connection(s)
I20260619 00:57:52.827159 19796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:52.834713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19796
I20260619 00:57:52.834955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:52.835368 388 external_mini_cluster.cc:1428] Setting key c44553afa4c9b8bc41503aa25f4f53b8
Jun 19 00:57:52 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:52.847970 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:52.830001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36525 (local address 127.0.97.62:40045)
0619 00:57:52.830544 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:52.830549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:52.831436 (+ 887us) server_negotiation.cc:407] Connection header received
0619 00:57:52.833375 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:52.833381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:52.833471 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:52.833591 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:52.835253 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.836133 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.837748 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.838145 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.838304 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:52.842032 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:52.842037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:52.842040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:52.842073 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:52.844615 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:52.845477 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:52.845482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:52.845484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:52.845606 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:52.846121 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:52.846124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:52.846126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:52.846408 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:52.846527 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:52.847512 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 00:57:52.847695 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":294,"threads_started":1}
I20260619 00:57:52.849501 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:52.830556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40045 (local address 127.0.97.1:36525)
0619 00:57:52.831267 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:52.831312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:52.833097 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:52.833840 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:52.833850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:52.834415 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:52.835061 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:52.835097 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.836287 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.836294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:52.837594 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:52.837606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:52.838410 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:52.838417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:52.838641 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:52.839350 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:52.839372 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:52.841809 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:52.844841 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:52.844853 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:52.844872 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:52.845331 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:52.845719 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:52.845727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:52.845733 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:52.845981 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:52.846540 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:52.846552 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:52.847040 (+ 488us) client_negotiation.cc:769] Sending connection context
0619 00:57:52.847351 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 00:57:52.847949 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260619 00:57:52.851169 19924 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40045
I20260619 00:57:52.851500 19924 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:52.852368 19924 heartbeater.cc:511] Master 127.0.97.62:40045 requested a full tablet report, sending...
I20260619 00:57:52.854615 19735 ts_manager.cc:194] Registered new tserver with Master: 315819b7b4d547e6888ba166e384b522 (127.0.97.1:44471)
I20260619 00:57:52.856256 19735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36525
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:57:52.892460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:53.235159 19930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:53.235438 19930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:53.235616 19930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:53.242789 19930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:53.242904 19930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:53.243070 19930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:53.252738 19930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:53.254109 19930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:53.255462 19930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:53.268971 19935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:53.268906 19936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:53.268975 19938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:53.269779 19930 server_base.cc:1034] running on GCE node
I20260619 00:57:53.270323 19930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:53.271207 19930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:53.272415 19930 hybrid_clock.cc:648] HybridClock initialized: now 1781830673272371 us; error 32 us; skew 500 ppm
I20260619 00:57:53.272701 19930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:53 dist-test-slave-r7bh krb5kdc[19690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:53.279273 19930 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:57:53.280802 19930 webserver.cc:460] Webserver started at http://127.0.97.2:41989/ using document root <none> and password file <none>
I20260619 00:57:53.281546 19930 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:53.281675 19930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:53.281997 19930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:53.284926 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "8155252f44524382a55a517d5a55804a"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "d1b9a31be57822e407795cf7138b55f0"
server_key_iv: "6bf3ed752a050e4773fdbb4c6e415605"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.285686 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8155252f44524382a55a517d5a55804a"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "d1b9a31be57822e407795cf7138b55f0"
server_key_iv: "6bf3ed752a050e4773fdbb4c6e415605"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.290745 19930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:57:53.294248 19945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.295686 19930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:53.295885 19930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "8155252f44524382a55a517d5a55804a"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "d1b9a31be57822e407795cf7138b55f0"
server_key_iv: "6bf3ed752a050e4773fdbb4c6e415605"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.296090 19930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:53.323174 19930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:53.327930 19930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:53.328264 19930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:53.329370 19930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:53.330770 19930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:53.330852 19930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.331025 19930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:53.331082 19930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.363240 19930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36419
I20260619 00:57:53.363274 20057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36419 every 8 connection(s)
I20260619 00:57:53.364677 19930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:53.367690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 19930
I20260619 00:57:53.367887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:53.368484 388 external_mini_cluster.cc:1428] Setting key fb938931cf5208ce2d5376dd39a17fda
Jun 19 00:57:53 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:53.388777 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.367823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36685 (local address 127.0.97.62:40045)
0619 00:57:53.369806 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.369813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.370998 (+ 1185us) server_negotiation.cc:407] Connection header received
0619 00:57:53.372857 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.372862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.372961 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.373072 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:53.374571 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.375291 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.376843 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.377286 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.377416 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.381415 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:53.381424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:53.381431 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:53.381477 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:53.385229 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.385992 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.386000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.386006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.386172 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.386606 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.386613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.386619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.387011 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:53.387162 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.387980 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.388257 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1862,"thread_start_us":1756,"threads_started":1}
I20260619 00:57:53.389477 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.368621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40045 (local address 127.0.97.2:36685)
0619 00:57:53.370845 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:53.370891 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:53.372574 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:53.373312 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:53.373320 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:53.373842 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:53.374405 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.374423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.375446 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.375450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.376675 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.376691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.377522 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.377529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.377751 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.378451 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:53.378488 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:53.381170 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:53.385408 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:53.385415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:53.385429 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:53.385865 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:53.386291 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:53.386295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:53.386297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:53.386497 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:53.387144 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:53.387150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:53.387604 (+ 454us) client_negotiation.cc:769] Sending connection context
0619 00:57:53.387852 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 00:57:53.388261 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1953,"thread_start_us":1662,"threads_started":1}
I20260619 00:57:53.391145 20058 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40045
I20260619 00:57:53.391518 20058 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:53.392532 20058 heartbeater.cc:511] Master 127.0.97.62:40045 requested a full tablet report, sending...
I20260619 00:57:53.394351 19735 ts_manager.cc:194] Registered new tserver with Master: 8155252f44524382a55a517d5a55804a (127.0.97.2:36419)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:57:53.395251 19735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:57:53.426003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:53.780992 20064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:53.781267 20064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:53.781437 20064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:53.788592 20064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:53.788689 20064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:53.788864 20064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:53.798339 20064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:53.799647 20064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:53.800990 20064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:53.814172 20069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:53.814188 20072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:53.814296 20070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:53.814955 20064 server_base.cc:1034] running on GCE node
I20260619 00:57:53.815599 20064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:53.816540 20064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:53.817755 20064 hybrid_clock.cc:648] HybridClock initialized: now 1781830673817677 us; error 72 us; skew 500 ppm
I20260619 00:57:53.818133 20064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:53 dist-test-slave-r7bh krb5kdc[19690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:53.821276 20064 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:57:53.822669 20064 webserver.cc:460] Webserver started at http://127.0.97.3:36079/ using document root <none> and password file <none>
I20260619 00:57:53.823403 20064 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:53.823542 20064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:53.823840 20064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:53.826992 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "b6d22a848f59454b979ad1a9d4fcd6f0"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "39a752413307420571e9c79f129d5eff"
server_key_iv: "a414b6c364d8871e76692ab1bcb2ad1c"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.827863 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b6d22a848f59454b979ad1a9d4fcd6f0"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "39a752413307420571e9c79f129d5eff"
server_key_iv: "a414b6c364d8871e76692ab1bcb2ad1c"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.832777 20064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:57:53.836403 20079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.837715 20064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:57:53.837900 20064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "b6d22a848f59454b979ad1a9d4fcd6f0"
format_stamp: "Formatted at 2026-06-19 00:57:53 on dist-test-slave-r7bh"
server_key: "39a752413307420571e9c79f129d5eff"
server_key_iv: "a414b6c364d8871e76692ab1bcb2ad1c"
server_key_version: "encryptionkey@0"
I20260619 00:57:53.838088 20064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:53.852350 20064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:53.856355 20064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:53.856648 20064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:53.857618 20064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:53.859025 20064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:53.859126 20064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.859262 20064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:53.859336 20064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:53.860294 19924 heartbeater.cc:503] Master 127.0.97.62:40045 was elected leader, sending a full tablet report...
I20260619 00:57:53.893590 20064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40697
I20260619 00:57:53.893648 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40697 every 8 connection(s)
I20260619 00:57:53.894927 20064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:53.899744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20064
I20260619 00:57:53.899942 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:53.900524 388 external_mini_cluster.cc:1428] Setting key 138d786b192d682f5bc3edb538b774d5
Jun 19 00:57:53 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:57:53.916707 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.897521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34003 (local address 127.0.97.62:40045)
0619 00:57:53.898188 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.898199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.899202 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 00:57:53.901962 (+ 2760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.901968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.902079 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.902212 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:53.904145 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.905179 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.906887 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.907309 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.907444 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.911082 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:53.911087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:53.911090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:53.911125 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:53.913623 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.914365 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.914372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.914375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.914500 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.914937 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.914943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.914946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.915221 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:53.915372 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.916076 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.916309 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":372,"threads_started":1}
I20260619 00:57:53.917712 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.898145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40045 (local address 127.0.97.3:34003)
0619 00:57:53.898982 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:53.899031 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:53.901619 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:53.902482 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:53.902494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:53.903341 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:53.903956 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.903976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.905440 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.905451 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.906733 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.906746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.907538 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.907545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.907788 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.908495 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:57:53.908519 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:53.910854 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:53.913789 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:53.913798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:53.913819 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:53.914234 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:53.914611 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:57:53.914616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:57:53.914619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:57:53.914824 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:57:53.915356 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:53.915365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:53.915711 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 00:57:53.915989 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 00:57:53.916561 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":307,"threads_started":1}
I20260619 00:57:53.919147 20192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40045
I20260619 00:57:53.919659 20192 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:53.920469 20192 heartbeater.cc:511] Master 127.0.97.62:40045 requested a full tablet report, sending...
I20260619 00:57:53.921907 19735 ts_manager.cc:194] Registered new tserver with Master: b6d22a848f59454b979ad1a9d4fcd6f0 (127.0.97.3:40697)
I20260619 00:57:53.922708 19735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34003
I20260619 00:57:53.927963 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:53.944897 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.932083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.97.62:40045)
0619 00:57:53.932244 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.932248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.932758 (+ 510us) server_negotiation.cc:407] Connection header received
0619 00:57:53.933014 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.933018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.933081 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.933217 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:53.934686 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.935402 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.936672 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.937114 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.937207 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.939461 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:53.939466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:53.939469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:53.939500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:53.942192 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.943091 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.943095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.943097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.943222 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:53.943564 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:53.943567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:53.943569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:53.943871 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:53.944054 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.944415 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.944593 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:57:53.948796 19735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:53.955734 19735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:53.979664 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.968231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40697 (local address 127.0.0.1:52040)
0619 00:57:53.970179 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:53.970228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:53.970430 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:53.971364 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:53.971371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:53.971400 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:53.971858 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.971872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.974135 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.974140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.976603 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.976625 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.978571 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.978577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.978779 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.978861 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 00:57:53.978972 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:57:53.979184 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"mutex_wait_us":1,"thread_start_us":1552,"threads_started":1}
I20260619 00:57:53.981458 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.968595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.0.97.3:40697)
0619 00:57:53.969581 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.969595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.970358 (+ 763us) server_negotiation.cc:407] Connection header received
0619 00:57:53.970674 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.970682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.970958 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.972054 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:53.972340 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.973979 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.976890 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.978338 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.980504 (+ 2166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.980780 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.980891 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.981074 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":499,"threads_started":1}
I20260619 00:57:53.983459 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.969353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.97.1:44471)
0619 00:57:53.969951 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.969957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.970426 (+ 469us) server_negotiation.cc:407] Connection header received
0619 00:57:53.971276 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.971287 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.971538 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.971732 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:53.972682 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.975442 (+ 2760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.979070 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.980632 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.980748 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.982224 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.982914 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.983106 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":284,"threads_started":1}
I20260619 00:57:53.984195 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.968693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44471 (local address 127.0.0.1:54202)
0619 00:57:53.970289 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:53.970325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:53.970517 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:53.971999 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:53.972002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:53.972026 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:53.972467 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.972483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.975630 (+ 3147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.975635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.978888 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.978900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.982456 (+ 3556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.982463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.982670 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.982712 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 00:57:53.983688 (+ 976us) client_negotiation.cc:240] Negotiation successful
0619 00:57:53.983857 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":1910,"threads_started":1}
I20260619 00:57:53.986075 20127 tablet_service.cc:1467] Processing CreateTablet for tablet cfafd21c635c4a509de7e063fd73687d (DEFAULT_TABLE table=test-table [id=e91b27d02ac7477890dd54aff354b5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:53.987854 20127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfafd21c635c4a509de7e063fd73687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:53.988767 19859 tablet_service.cc:1467] Processing CreateTablet for tablet cfafd21c635c4a509de7e063fd73687d (DEFAULT_TABLE table=test-table [id=e91b27d02ac7477890dd54aff354b5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:53.990183 19859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfafd21c635c4a509de7e063fd73687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:53.992187 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.970937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36419 (local address 127.0.0.1:34836)
0619 00:57:53.979944 (+ 9007us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:53.979966 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:53.980072 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:53.985316 (+ 5244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:53.985320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:53.985351 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:53.985545 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.985562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.987525 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.987530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.989399 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:53.989427 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.991301 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.991308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:53.991508 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.991556 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 00:57:53.991668 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 00:57:53.991857 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8903,"thread_start_us":13783,"threads_started":1}
I20260619 00:57:53.992208 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:53.971220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.0.97.2:36419)
0619 00:57:53.984201 (+ 12981us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:53.984213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:53.984254 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:57:53.984795 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:53.984802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:53.985116 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:53.985328 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:53.985696 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.987325 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.989643 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:53.991091 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:53.991266 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:53.991481 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:53.991722 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 00:57:53.991902 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12745,"thread_start_us":13205,"threads_started":1}
I20260619 00:57:53.996948 19993 tablet_service.cc:1467] Processing CreateTablet for tablet cfafd21c635c4a509de7e063fd73687d (DEFAULT_TABLE table=test-table [id=e91b27d02ac7477890dd54aff354b5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:53.998538 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfafd21c635c4a509de7e063fd73687d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:54.002512 20211 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Bootstrap starting.
I20260619 00:57:54.004719 20212 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Bootstrap starting.
I20260619 00:57:54.006484 20211 tablet_bootstrap.cc:654] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:54.007838 20211 log.cc:824] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:54.008610 20212 tablet_bootstrap.cc:654] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:54.009958 20212 log.cc:824] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:54.011852 20211 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: No bootstrap required, opened a new log
I20260619 00:57:54.012142 20211 ts_tablet_manager.cc:1397] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 00:57:54.012732 20212 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: No bootstrap required, opened a new log
I20260619 00:57:54.013011 20212 ts_tablet_manager.cc:1397] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 00:57:54.017529 20215 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Bootstrap starting.
I20260619 00:57:54.021485 20212 raft_consensus.cc:348] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.021967 20212 raft_consensus.cc:374] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:54.022071 20212 raft_consensus.cc:729] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d22a848f59454b979ad1a9d4fcd6f0, State: Initialized, Role: FOLLOWER
I20260619 00:57:54.022462 20215 tablet_bootstrap.cc:654] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:54.022750 20212 consensus_queue.cc:260] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.022917 20211 raft_consensus.cc:348] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.023376 20211 raft_consensus.cc:374] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:54.023541 20211 raft_consensus.cc:729] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315819b7b4d547e6888ba166e384b522, State: Initialized, Role: FOLLOWER
I20260619 00:57:54.023885 20215 log.cc:824] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:54.024121 20192 heartbeater.cc:503] Master 127.0.97.62:40045 was elected leader, sending a full tablet report...
I20260619 00:57:54.024255 20211 consensus_queue.cc:260] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.025544 20212 ts_tablet_manager.cc:1428] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 00:57:54.027174 20211 ts_tablet_manager.cc:1428] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260619 00:57:54.027796 20215 tablet_bootstrap.cc:492] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: No bootstrap required, opened a new log
I20260619 00:57:54.028093 20215 ts_tablet_manager.cc:1397] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 00:57:54.036433 20215 raft_consensus.cc:348] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.036844 20215 raft_consensus.cc:374] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:54.036962 20215 raft_consensus.cc:729] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8155252f44524382a55a517d5a55804a, State: Initialized, Role: FOLLOWER
I20260619 00:57:54.037506 20215 consensus_queue.cc:260] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.038476 20058 heartbeater.cc:503] Master 127.0.97.62:40045 was elected leader, sending a full tablet report...
I20260619 00:57:54.039170 20215 ts_tablet_manager.cc:1428] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260619 00:57:54.053293 20216 raft_consensus.cc:482] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:54.053563 20216 raft_consensus.cc:504] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.055171 20216 leader_election.cc:288] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315819b7b4d547e6888ba166e384b522 (127.0.97.1:44471), 8155252f44524382a55a517d5a55804a (127.0.97.2:36419)
I20260619 00:57:54.062381 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.055600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47173 (local address 127.0.97.1:44471)
0619 00:57:54.055840 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.055846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.055868 (+ 22us) server_negotiation.cc:407] Connection header received
0619 00:57:54.056419 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.056430 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.056495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.056572 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:54.057083 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.058116 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.060415 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.061578 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.061722 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.061829 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.062019 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.062155 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:57:54.062413 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.055483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44471 (local address 127.0.97.3:47173)
0619 00:57:54.055699 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:54.055717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:54.055840 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:54.056625 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:54.056633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:54.056668 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:54.056944 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:54.056955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.058244 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.058247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:54.060259 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:54.060275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.061728 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.061732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:54.061863 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.061895 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:57:54.061969 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 00:57:54.062114 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 00:57:54.062891 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.056231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58769 (local address 127.0.97.2:36419)
0619 00:57:54.056463 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.056469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.056703 (+ 234us) server_negotiation.cc:407] Connection header received
0619 00:57:54.056945 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.056948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.057018 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.057115 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:54.057566 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.058466 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.060690 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.061937 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.062082 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.062166 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.062515 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.062654 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:57:54.062923 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.055930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36419 (local address 127.0.97.3:58769)
0619 00:57:54.056591 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:54.056608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:54.056770 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:54.057130 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:54.057133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:54.057153 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:54.057423 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:54.057430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.058595 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.058600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:54.060553 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:54.060563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.062151 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.062156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:54.062325 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.062354 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:57:54.062465 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:57:54.062594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":2,"spinlock_wait_cycles":36096,"thread_start_us":301,"threads_started":1}
I20260619 00:57:54.063825 19879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfafd21c635c4a509de7e063fd73687d" candidate_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315819b7b4d547e6888ba166e384b522" is_pre_election: true
I20260619 00:57:54.064167 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfafd21c635c4a509de7e063fd73687d" candidate_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8155252f44524382a55a517d5a55804a" is_pre_election: true
I20260619 00:57:54.064479 19879 raft_consensus.cc:2454] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d22a848f59454b979ad1a9d4fcd6f0 in term 0.
I20260619 00:57:54.064724 20013 raft_consensus.cc:2454] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d22a848f59454b979ad1a9d4fcd6f0 in term 0.
I20260619 00:57:54.065232 20083 leader_election.cc:302] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315819b7b4d547e6888ba166e384b522, b6d22a848f59454b979ad1a9d4fcd6f0; no voters:
I20260619 00:57:54.065625 20216 raft_consensus.cc:2781] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:54.065785 20216 raft_consensus.cc:482] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:54.065900 20216 raft_consensus.cc:3037] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:54.068145 20216 raft_consensus.cc:504] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.068791 20216 leader_election.cc:288] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [CANDIDATE]: Term 1 election: Requested vote from peers 315819b7b4d547e6888ba166e384b522 (127.0.97.1:44471), 8155252f44524382a55a517d5a55804a (127.0.97.2:36419)
I20260619 00:57:54.069267 19879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfafd21c635c4a509de7e063fd73687d" candidate_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315819b7b4d547e6888ba166e384b522"
I20260619 00:57:54.069514 19879 raft_consensus.cc:3037] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:54.069646 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfafd21c635c4a509de7e063fd73687d" candidate_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8155252f44524382a55a517d5a55804a"
I20260619 00:57:54.069886 20013 raft_consensus.cc:3037] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:54.071822 19879 raft_consensus.cc:2454] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d22a848f59454b979ad1a9d4fcd6f0 in term 1.
I20260619 00:57:54.072129 20013 raft_consensus.cc:2454] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d22a848f59454b979ad1a9d4fcd6f0 in term 1.
I20260619 00:57:54.072192 20083 leader_election.cc:302] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315819b7b4d547e6888ba166e384b522, b6d22a848f59454b979ad1a9d4fcd6f0; no voters:
I20260619 00:57:54.072537 20216 raft_consensus.cc:2781] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:54.073020 20216 raft_consensus.cc:686] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 1 LEADER]: Becoming Leader. State: Replica: b6d22a848f59454b979ad1a9d4fcd6f0, State: Running, Role: LEADER
I20260619 00:57:54.073508 20216 consensus_queue.cc:237] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } }
I20260619 00:57:54.078495 19734 catalog_manager.cc:5495] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6d22a848f59454b979ad1a9d4fcd6f0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d22a848f59454b979ad1a9d4fcd6f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 } health_report { overall_health: UNKNOWN } } }
W20260619 00:57:54.079620 19925 tablet.cc:2369] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:54.105201 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.100188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.97.3:40697)
0619 00:57:54.100383 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.100387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.100445 (+ 58us) server_negotiation.cc:407] Connection header received
0619 00:57:54.100646 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.100651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.100769 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.100942 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:54.101264 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.101959 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.103265 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.103651 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.103750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.104178 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:54.104684 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:54.104800 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.104939 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.105002 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:57:54.113560 20013 raft_consensus.cc:1261] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 1 FOLLOWER]: Refusing update from remote peer b6d22a848f59454b979ad1a9d4fcd6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:54.113633 19879 raft_consensus.cc:1261] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 1 FOLLOWER]: Refusing update from remote peer b6d22a848f59454b979ad1a9d4fcd6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:54.114611 20221 consensus_queue.cc:1035] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8155252f44524382a55a517d5a55804a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:54.115015 20216 consensus_queue.cc:1035] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315819b7b4d547e6888ba166e384b522" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:57:54.117410 20059 tablet.cc:2369] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:54.127260 20225 mvcc.cc:205] Tried to move back new op lower bound from 7298378441159053312 to 7298378441009065984. Current Snapshot: MvccSnapshot[applied={T|T < 7298378441159053312}]
I20260619 00:57:54.144874 19734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:54.145211 19734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:54.149034 19734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de9683b22a3f4a67863dc6157e306335: Sending DeleteTablet for 3 replicas of tablet cfafd21c635c4a509de7e063fd73687d
I20260619 00:57:54.150300 19859 tablet_service.cc:1514] Processing DeleteTablet for tablet cfafd21c635c4a509de7e063fd73687d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54202
I20260619 00:57:54.150301 20127 tablet_service.cc:1514] Processing DeleteTablet for tablet cfafd21c635c4a509de7e063fd73687d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52040
I20260619 00:57:54.151230 20233 tablet_replica.cc:330] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: stopping tablet replica
I20260619 00:57:54.151459 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet cfafd21c635c4a509de7e063fd73687d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34836
I20260619 00:57:54.151846 20233 raft_consensus.cc:2229] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:54.152410 20233 raft_consensus.cc:2258] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:54.153441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19796
I20260619 00:57:54.153486 20234 tablet_replica.cc:330] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: stopping tablet replica
I20260619 00:57:54.153992 20234 raft_consensus.cc:2229] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:54.154670 20234 raft_consensus.cc:2258] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:54.156458 20233 ts_tablet_manager.cc:1905] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:54.157089 20234 ts_tablet_manager.cc:1905] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:54.163440 20233 ts_tablet_manager.cc:1918] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:54.163630 20233 log.cc:1196] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-0/wal/wals/cfafd21c635c4a509de7e063fd73687d
I20260619 00:57:54.164281 20233 ts_tablet_manager.cc:1939] T cfafd21c635c4a509de7e063fd73687d P 315819b7b4d547e6888ba166e384b522: Deleting consensus metadata
I20260619 00:57:54.165119 20234 ts_tablet_manager.cc:1918] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:54.166695 19722 catalog_manager.cc:4841] TS 315819b7b4d547e6888ba166e384b522 (127.0.97.1:44471): tablet cfafd21c635c4a509de7e063fd73687d (table test-table [id=e91b27d02ac7477890dd54aff354b5c8]) successfully deleted
I20260619 00:57:54.167449 20235 tablet_replica.cc:330] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: stopping tablet replica
I20260619 00:57:54.167843 20235 raft_consensus.cc:2229] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:54.168334 20235 raft_consensus.cc:2258] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:54.168368 20234 log.cc:1196] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/cfafd21c635c4a509de7e063fd73687d
I20260619 00:57:54.168972 20234 ts_tablet_manager.cc:1939] T cfafd21c635c4a509de7e063fd73687d P b6d22a848f59454b979ad1a9d4fcd6f0: Deleting consensus metadata
I20260619 00:57:54.170833 19719 catalog_manager.cc:4841] TS b6d22a848f59454b979ad1a9d4fcd6f0 (127.0.97.3:40697): tablet cfafd21c635c4a509de7e063fd73687d (table test-table [id=e91b27d02ac7477890dd54aff354b5c8]) successfully deleted
I20260619 00:57:54.170936 20235 ts_tablet_manager.cc:1905] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 00:57:54 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:57:54.177989 20235 ts_tablet_manager.cc:1918] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:54.178215 20235 log.cc:1196] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/cfafd21c635c4a509de7e063fd73687d
I20260619 00:57:54.178820 20235 ts_tablet_manager.cc:1939] T cfafd21c635c4a509de7e063fd73687d P 8155252f44524382a55a517d5a55804a: Deleting consensus metadata
I20260619 00:57:54.180737 19722 catalog_manager.cc:4841] TS 8155252f44524382a55a517d5a55804a (127.0.97.2:36419): tablet cfafd21c635c4a509de7e063fd73687d (table test-table [id=e91b27d02ac7477890dd54aff354b5c8]) successfully deleted
I20260619 00:57:54.185216 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.155718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.0.97.1:44471)
0619 00:57:54.155937 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.155946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.166067 (+ 10121us) server_negotiation.cc:407] Connection header received
0619 00:57:54.166355 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.166361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.166454 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.166525 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:54.168328 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.169417 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.171278 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.171925 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.172050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.177727 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:54.177746 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:54.177766 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:54.177829 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:54.182007 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.182861 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.182871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.182875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.183045 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.183454 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.183460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.183464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.183942 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:54.184254 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.184553 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.184795 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 00:57:54.185967 19919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:54.278425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19796
I20260619 00:57:54.302428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19930
Jun 19 00:57:54 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:57:54.318346 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.303286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.97.2:36419)
0619 00:57:54.303476 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.303483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.303506 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:57:54.303711 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.303717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.303802 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.303911 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:54.305285 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.306052 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.307324 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.307751 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.307881 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.312495 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:54.312510 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:54.312527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:54.312562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:54.315417 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.316175 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.316183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.316185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.316317 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.316690 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.316694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.316697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.317145 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:54.317478 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.317759 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.317979 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:57:54.319124 20053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:54.409796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19930
I20260619 00:57:54.427872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20064
Jun 19 00:57:54 dist-test-slave-r7bh krb5kdc[19690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:57:54.444660 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:54.428649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.0.97.3:40697)
0619 00:57:54.428846 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:54.428853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:54.428949 (+ 96us) server_negotiation.cc:407] Connection header received
0619 00:57:54.429148 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:54.429154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:54.429248 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:54.429353 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:54.430614 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.431472 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.432742 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:54.433312 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:54.433452 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:54.438127 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:54.438138 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:54.438153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:54.438189 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:54.441547 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.442299 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.442307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.442309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.442510 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:54.442916 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:54.442925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:54.442929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:54.443305 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:54.443633 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:54.443935 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 00:57:54.444153 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:57:54.445550 20187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:54.573312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20064
I20260619 00:57:54.593509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 19703
I20260619 00:57:54.594646 19765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:54.705794 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 19703
2026-06-19T00:57:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4349 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T00:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:54Z Disabled control of system clock
I20260619 00:57:54.785512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46065
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:55.130611 20244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:55.130934 20244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:55.131111 20244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:55.138636 20244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:55.138763 20244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:55.138862 20244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:55.138984 20244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:55.139070 20244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:55.149446 20244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:55.150838 20244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:55.152882 20244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:55.165104 20249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:55.165031 20252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:55.165078 20250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:55.165861 20244 server_base.cc:1034] running on GCE node
I20260619 00:57:55.166673 20244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:55.168041 20244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:55.169265 20244 hybrid_clock.cc:648] HybridClock initialized: now 1781830675169222 us; error 62 us; skew 500 ppm
I20260619 00:57:55.169626 20244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:55.171942 20244 webserver.cc:460] Webserver started at http://127.0.97.62:46757/ using document root <none> and password file <none>
I20260619 00:57:55.172623 20244 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:55.172760 20244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:55.173097 20244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:55.175966 20244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "743da4bb59554a428189dff6f26a85d7"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "643f7418a71d3074e14b9605330e8fce"
server_key_iv: "4325b845dd3ae4691214f41cc46aa06f"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.176743 20244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "743da4bb59554a428189dff6f26a85d7"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "643f7418a71d3074e14b9605330e8fce"
server_key_iv: "4325b845dd3ae4691214f41cc46aa06f"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.181926 20244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 00:57:55.185334 20258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:55.186723 20244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:55.186923 20244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "743da4bb59554a428189dff6f26a85d7"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "643f7418a71d3074e14b9605330e8fce"
server_key_iv: "4325b845dd3ae4691214f41cc46aa06f"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.187148 20244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:55.207237 20244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:55.208098 20244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:55.208392 20244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:55.227752 20244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46065
I20260619 00:57:55.227777 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46065 every 8 connection(s)
I20260619 00:57:55.229156 20244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:55.233058 20310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:55.233596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20244
I20260619 00:57:55.233799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:55.234229 388 external_mini_cluster.cc:1428] Setting key 4e155e328d371a5ecb61bc2f1924a5e4
I20260619 00:57:55.247980 20310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: Bootstrap starting.
I20260619 00:57:55.250006 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:55.236135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.0.97.62:46065)
0619 00:57:55.237225 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:55.237239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:55.237298 (+ 59us) server_negotiation.cc:407] Connection header received
0619 00:57:55.239853 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:55.239883 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:55.240520 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:55.241024 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:55.241547 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.243407 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.244747 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.245870 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.246042 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:55.247183 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:55.247193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:55.247210 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:55.247922 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:55.248027 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:55.248054 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:55.248177 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:55.248354 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 00:57:55.248683 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":303,"threads_started":1}
I20260619 00:57:55.251962 20310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:55.253129 20310 log.cc:824] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:55.255591 20310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: No bootstrap required, opened a new log
I20260619 00:57:55.262852 20310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260619 00:57:55.263207 20310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:55.263320 20310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743da4bb59554a428189dff6f26a85d7, State: Initialized, Role: FOLLOWER
I20260619 00:57:55.263842 20310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260619 00:57:55.264063 20310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:55.264202 20310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:55.264353 20310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:55.266497 20310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260619 00:57:55.266974 20310 leader_election.cc:302] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743da4bb59554a428189dff6f26a85d7; no voters:
I20260619 00:57:55.267532 20310 leader_election.cc:288] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:55.267688 20315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:55.268368 20315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [term 1 LEADER]: Becoming Leader. State: Replica: 743da4bb59554a428189dff6f26a85d7, State: Running, Role: LEADER
I20260619 00:57:55.268951 20315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } }
I20260619 00:57:55.269731 20310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:55.271942 20317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743da4bb59554a428189dff6f26a85d7. Latest consensus state: current_term: 1 leader_uuid: "743da4bb59554a428189dff6f26a85d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } } }
I20260619 00:57:55.272433 20317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:55.272075 20316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743da4bb59554a428189dff6f26a85d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743da4bb59554a428189dff6f26a85d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46065 } } }
I20260619 00:57:55.272713 20316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7 [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:55.274189 20324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:55.279356 20324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:55.288192 20324 catalog_manager.cc:1347] Generated new cluster ID: ed64d700c9b44f2e92d9b94a5fb5a00c
I20260619 00:57:55.288354 20324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:55.302016 20324 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:55.303323 20324 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:55.316187 20324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: Generated new TSK 0
I20260619 00:57:55.317137 20324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:57:55.321944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--builtin_ntp_servers=127.0.97.20:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 00:57:55.666644 20334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:55.666946 20334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:55.667102 20334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:55.674527 20334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:55.674633 20334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:55.674803 20334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:57:55.683995 20334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:55.685336 20334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:55.687192 20334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:55.700989 20339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:55.701093 20340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:55.701122 20342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:55.701709 20334 server_base.cc:1034] running on GCE node
I20260619 00:57:55.702365 20334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:55.703434 20334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:55.704674 20334 hybrid_clock.cc:648] HybridClock initialized: now 1781830675704589 us; error 77 us; skew 500 ppm
I20260619 00:57:55.705039 20334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:55.707391 20334 webserver.cc:460] Webserver started at http://127.0.97.1:38893/ using document root <none> and password file <none>
I20260619 00:57:55.708109 20334 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:55.708227 20334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:55.708554 20334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:55.711499 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data/instance:
uuid: "77e7fca2103442dd9cd65b1c9ea44d0d"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "4f5729e6a13adfd9f646b5b0af6873d8"
server_key_iv: "e43f0d7eabb5e2104e375d8624f8bd3d"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.712333 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77e7fca2103442dd9cd65b1c9ea44d0d"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "4f5729e6a13adfd9f646b5b0af6873d8"
server_key_iv: "e43f0d7eabb5e2104e375d8624f8bd3d"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.717525 20334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 00:57:55.721230 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:55.722545 20334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:57:55.722748 20334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal
uuid: "77e7fca2103442dd9cd65b1c9ea44d0d"
format_stamp: "Formatted at 2026-06-19 00:57:55 on dist-test-slave-r7bh"
server_key: "4f5729e6a13adfd9f646b5b0af6873d8"
server_key_iv: "e43f0d7eabb5e2104e375d8624f8bd3d"
server_key_version: "encryptionkey@0"
I20260619 00:57:55.722961 20334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:55.739850 20334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:55.740890 20334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:55.741221 20334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:55.742272 20334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:55.743906 20334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:55.744036 20334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:55.744156 20334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:55.744241 20334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:55.776134 20334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38545
I20260619 00:57:55.776146 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38545 every 8 connection(s)
I20260619 00:57:55.777498 20334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:57:55.781472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20334
I20260619 00:57:55.781646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:57:55.782131 388 external_mini_cluster.cc:1428] Setting key 657d03cc8b10f5f3dc6c9f9a854259f2
I20260619 00:57:55.784618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--builtin_ntp_servers=127.0.97.20:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:55.790341 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:55.780033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54933 (local address 127.0.97.62:46065)
0619 00:57:55.780474 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:55.780479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:55.781825 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 00:57:55.783891 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:55.783896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:55.783988 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:55.784107 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:55.785284 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.785998 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.787556 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.787964 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.788095 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:55.789482 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:55.789489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:55.789493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:55.789520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:55.789558 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:55.789570 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:55.789672 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:55.789898 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 00:57:55.789993 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260619 00:57:55.791409 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:55.780652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46065 (local address 127.0.97.1:54933)
0619 00:57:55.781634 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:55.781696 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:55.783613 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:55.784349 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:55.784357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:55.784448 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:55.785120 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:55.785138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.786171 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.786175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:55.787395 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:55.787407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:55.788205 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:55.788212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:55.788436 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:55.789177 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:55.789179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:55.789220 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:55.789294 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:55.789671 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:55.789683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:55.789693 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 00:57:55.789875 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 00:57:55.790276 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
I20260619 00:57:55.793160 20461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46065
I20260619 00:57:55.793457 20461 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:55.794195 20461 heartbeater.cc:511] Master 127.0.97.62:46065 requested a full tablet report, sending...
I20260619 00:57:55.796341 20275 ts_manager.cc:194] Registered new tserver with Master: 77e7fca2103442dd9cd65b1c9ea44d0d (127.0.97.1:38545)
I20260619 00:57:55.797847 20275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54933
W20260619 00:57:56.114095 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:56.114446 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:56.114615 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:56.121881 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:56.122005 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:56.122174 20465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:57:56.131311 20465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:56.132623 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:56.134436 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:56.147600 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:56.147678 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:56.147645 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:56.148677 20465 server_base.cc:1034] running on GCE node
I20260619 00:57:56.149431 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:56.150602 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:56.151856 20465 hybrid_clock.cc:648] HybridClock initialized: now 1781830676151809 us; error 44 us; skew 500 ppm
I20260619 00:57:56.152201 20465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:56.154603 20465 webserver.cc:460] Webserver started at http://127.0.97.2:40881/ using document root <none> and password file <none>
I20260619 00:57:56.155349 20465 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:56.155479 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:56.155818 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:56.158795 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data/instance:
uuid: "513ebe26dee44defae8963cc6b856b36"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "4c78c21bc359520f3b777a6431b6d5ce"
server_key_iv: "debfedb5d2bee7a2ef4163cad4f61f14"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.159610 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal/instance:
uuid: "513ebe26dee44defae8963cc6b856b36"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "4c78c21bc359520f3b777a6431b6d5ce"
server_key_iv: "debfedb5d2bee7a2ef4163cad4f61f14"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.164712 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 00:57:56.168177 20479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.169622 20465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:57:56.169811 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal
uuid: "513ebe26dee44defae8963cc6b856b36"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "4c78c21bc359520f3b777a6431b6d5ce"
server_key_iv: "debfedb5d2bee7a2ef4163cad4f61f14"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.170017 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:56.187877 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:56.188768 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:56.189077 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:56.190037 20465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:56.191588 20465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:56.191677 20465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.191834 20465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:56.191898 20465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.227401 20465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43523
I20260619 00:57:56.227452 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43523 every 8 connection(s)
I20260619 00:57:56.228960 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:57:56.229489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20465
I20260619 00:57:56.229661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:57:56.230154 388 external_mini_cluster.cc:1428] Setting key 6652e831e9737825115d504e1b9cffe4
I20260619 00:57:56.233328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--builtin_ntp_servers=127.0.97.20:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 00:57:56.250159 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.233980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33209 (local address 127.0.97.62:46065)
0619 00:57:56.234382 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.234392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.236259 (+ 1867us) server_negotiation.cc:407] Connection header received
0619 00:57:56.240047 (+ 3788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.240054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.240178 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.240270 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:56.242500 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.243823 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.246308 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.247092 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.247237 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.249135 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:56.249145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:56.249153 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:56.249181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:56.249223 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:56.249244 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:56.249334 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.249610 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.249727 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260619 00:57:56.251665 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.233246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46065 (local address 127.0.97.2:33209)
0619 00:57:56.236085 (+ 2839us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:56.236138 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:56.239618 (+ 3480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:56.241007 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:56.241023 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:56.241194 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:56.242244 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.242275 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.244000 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.244007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.246094 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.246112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.247469 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.247480 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.247826 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.248690 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:56.248695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:56.248749 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:56.248885 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:56.249354 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:56.249365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:56.249374 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 00:57:56.249565 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 00:57:56.249989 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2208,"thread_start_us":1947,"threads_started":1}
I20260619 00:57:56.253343 20592 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46065
I20260619 00:57:56.253661 20592 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:56.254598 20592 heartbeater.cc:511] Master 127.0.97.62:46065 requested a full tablet report, sending...
I20260619 00:57:56.256263 20275 ts_manager.cc:194] Registered new tserver with Master: 513ebe26dee44defae8963cc6b856b36 (127.0.97.2:43523)
I20260619 00:57:56.257213 20275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33209
W20260619 00:57:56.567584 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:56.567899 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:56.568054 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:56.575953 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:56.576077 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:56.576244 20595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:57:56.585912 20595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:56.587262 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:56.589092 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:56.602348 20600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:56.602376 20603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:56.602527 20601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:56.603189 20595 server_base.cc:1034] running on GCE node
I20260619 00:57:56.603824 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:56.604858 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:56.606108 20595 hybrid_clock.cc:648] HybridClock initialized: now 1781830676606034 us; error 51 us; skew 500 ppm
I20260619 00:57:56.606451 20595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 00:57:56.608903 20595 webserver.cc:460] Webserver started at http://127.0.97.3:36583/ using document root <none> and password file <none>
I20260619 00:57:56.609629 20595 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:56.609757 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:56.610080 20595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:56.613086 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data/instance:
uuid: "80e7a371a8814905b8d458ae17231926"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "e0273fd0ad10865014fd9a35ee8d3c51"
server_key_iv: "8567e5c61185c3a74e847687ba5aece5"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.613873 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal/instance:
uuid: "80e7a371a8814905b8d458ae17231926"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "e0273fd0ad10865014fd9a35ee8d3c51"
server_key_iv: "8567e5c61185c3a74e847687ba5aece5"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.619113 20595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:57:56.622588 20609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.624030 20595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:57:56.624226 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal
uuid: "80e7a371a8814905b8d458ae17231926"
format_stamp: "Formatted at 2026-06-19 00:57:56 on dist-test-slave-r7bh"
server_key: "e0273fd0ad10865014fd9a35ee8d3c51"
server_key_iv: "8567e5c61185c3a74e847687ba5aece5"
server_key_version: "encryptionkey@0"
I20260619 00:57:56.624495 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:56.646159 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:56.647017 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:56.647306 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:56.648290 20595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:57:56.649866 20595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:57:56.649995 20595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.650136 20595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:57:56.650226 20595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:56.682714 20595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39539
I20260619 00:57:56.682773 20721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39539 every 8 connection(s)
I20260619 00:57:56.683959 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:57:56.694002 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20595
I20260619 00:57:56.694180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:57:56.694738 388 external_mini_cluster.cc:1428] Setting key ca0d15fa873aac7a3ed7b01fc4a7167b
I20260619 00:57:56.696540 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.686579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41953 (local address 127.0.97.62:46065)
0619 00:57:56.686765 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.686769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.688060 (+ 1291us) server_negotiation.cc:407] Connection header received
0619 00:57:56.689901 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.689905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.689970 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.690047 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:56.691125 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.691953 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.693488 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.694112 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.694274 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.695685 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:56.695691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:56.695700 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:56.695717 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:56.695748 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:56.695761 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:56.695837 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.696084 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.696183 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:57:56.697703 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.686968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46065 (local address 127.0.97.3:41953)
0619 00:57:56.687903 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:56.687940 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:56.689638 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:56.690260 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:56.690268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:56.690351 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:57:56.690991 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.691009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.692059 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.692062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.693373 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.693385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.694367 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.694372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.694670 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.695396 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 00:57:56.695399 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:57:56.695437 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 00:57:56.695515 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:57:56.695862 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:57:56.695874 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:57:56.695882 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 00:57:56.696072 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 00:57:56.696466 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":307,"threads_started":1}
I20260619 00:57:56.698906 20722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46065
I20260619 00:57:56.699199 20722 heartbeater.cc:464] Registering TS with master...
I20260619 00:57:56.699919 20722 heartbeater.cc:511] Master 127.0.97.62:46065 requested a full tablet report, sending...
I20260619 00:57:56.701232 20275 ts_manager.cc:194] Registered new tserver with Master: 80e7a371a8814905b8d458ae17231926 (127.0.97.3:39539)
I20260619 00:57:56.701952 20275 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:41953
I20260619 00:57:56.710021 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:57:56.719756 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.714102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.97.62:46065)
0619 00:57:56.714260 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.714265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.714718 (+ 453us) server_negotiation.cc:407] Connection header received
0619 00:57:56.714867 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.714870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.714944 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.715027 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:56.715551 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.716256 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.717695 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.718107 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.718202 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.719188 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:56.719192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:56.719195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:56.719213 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:56.719259 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:56.719271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:56.719340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.719448 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.719521 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 00:57:56.723436 20275 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:57:56.730643 20275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:57:56.755081 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.742674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55540)
0619 00:57:56.743762 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:56.743825 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:56.744039 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:56.747420 (+ 3381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:56.747429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:56.747463 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:56.747925 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.747939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.749827 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.749835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.752134 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.752167 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.754131 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.754137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.754335 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.754390 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 00:57:56.754527 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 00:57:56.754721 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":4421,"threads_started":1}
I20260619 00:57:56.755944 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.744674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43523 (local address 127.0.0.1:35590)
0619 00:57:56.747395 (+ 2721us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:56.747411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:56.747508 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:56.748260 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:56.748264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:56.748289 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:56.748790 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.748800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.750856 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.750861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.753405 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.753417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.755143 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.755152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.755330 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.755359 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:57:56.755422 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 00:57:56.755547 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2577,"mutex_wait_us":124,"spinlock_wait_cycles":2304,"thread_start_us":2112,"threads_started":1}
I20260619 00:57:56.756174 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.742904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.97.1:38545)
0619 00:57:56.746177 (+ 3273us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.746189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.746252 (+ 63us) server_negotiation.cc:407] Connection header received
0619 00:57:56.746393 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.746406 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.746759 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.748140 (+ 1381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:56.748392 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.749693 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.752398 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.753937 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.754087 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.754302 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.754587 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.754759 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3071,"thread_start_us":2951,"threads_started":1}
I20260619 00:57:56.756881 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.745334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.0.97.2:43523)
0619 00:57:56.746596 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.746608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.747643 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 00:57:56.747744 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.747752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.747986 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.748183 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:56.749014 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.750685 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.753590 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.754808 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.754929 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.756284 (+ 1355us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.756350 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.756509 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":862,"threads_started":1}
I20260619 00:57:56.757395 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.743485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.0.97.3:39539)
0619 00:57:56.745664 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:56.745688 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:56.747344 (+ 1656us) server_negotiation.cc:407] Connection header received
0619 00:57:56.747451 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:56.747459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:56.747724 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:56.747942 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:56.749105 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.750708 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.754384 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.755762 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.755912 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.756185 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:56.756709 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 00:57:56.756899 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1761,"threads_started":1}
I20260619 00:57:56.758008 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:56.742883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39539 (local address 127.0.0.1:55408)
0619 00:57:56.747076 (+ 4193us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:56.747110 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:56.747268 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:56.748289 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:56.748293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:56.748332 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:56.748909 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.748922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.750857 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.750861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.753895 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:56.753910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:56.755959 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:56.755969 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:56.756510 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:56.756542 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:57:56.756623 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 00:57:56.756745 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4047,"mutex_wait_us":406,"spinlock_wait_cycles":5632,"thread_start_us":308,"threads_started":1}
I20260619 00:57:56.759827 20396 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec (DEFAULT_TABLE table=test-table [id=ab37b4e1b62b4eb58351bc0f08b73c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:56.760165 20527 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec (DEFAULT_TABLE table=test-table [id=ab37b4e1b62b4eb58351bc0f08b73c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:56.761673 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:56.761693 20527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:56.763876 20657 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec (DEFAULT_TABLE table=test-table [id=ab37b4e1b62b4eb58351bc0f08b73c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:57:56.765297 20657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:56.776976 20740 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Bootstrap starting.
I20260619 00:57:56.781224 20740 tablet_bootstrap.cc:654] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:56.782662 20740 log.cc:824] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:56.787557 20741 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Bootstrap starting.
I20260619 00:57:56.788971 20743 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Bootstrap starting.
I20260619 00:57:56.791736 20741 tablet_bootstrap.cc:654] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:56.793053 20743 tablet_bootstrap.cc:654] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:56.793036 20741 log.cc:824] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:56.793499 20740 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: No bootstrap required, opened a new log
I20260619 00:57:56.793910 20740 ts_tablet_manager.cc:1397] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260619 00:57:56.794246 20743 log.cc:824] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:56.797233 20743 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: No bootstrap required, opened a new log
I20260619 00:57:56.797377 20741 tablet_bootstrap.cc:492] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: No bootstrap required, opened a new log
I20260619 00:57:56.797494 20743 ts_tablet_manager.cc:1397] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 00:57:56.797657 20741 ts_tablet_manager.cc:1397] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 00:57:56.800686 20461 heartbeater.cc:503] Master 127.0.97.62:46065 was elected leader, sending a full tablet report...
I20260619 00:57:56.806275 20741 raft_consensus.cc:348] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.806684 20741 raft_consensus.cc:374] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:56.806778 20741 raft_consensus.cc:729] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e7a371a8814905b8d458ae17231926, State: Initialized, Role: FOLLOWER
I20260619 00:57:56.806529 20740 raft_consensus.cc:348] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.806965 20740 raft_consensus.cc:374] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:56.807081 20740 raft_consensus.cc:729] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e7fca2103442dd9cd65b1c9ea44d0d, State: Initialized, Role: FOLLOWER
I20260619 00:57:56.807550 20741 consensus_queue.cc:260] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.807677 20740 consensus_queue.cc:260] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.808266 20743 raft_consensus.cc:348] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.808708 20722 heartbeater.cc:503] Master 127.0.97.62:46065 was elected leader, sending a full tablet report...
I20260619 00:57:56.808876 20743 raft_consensus.cc:374] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:56.809005 20743 raft_consensus.cc:729] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513ebe26dee44defae8963cc6b856b36, State: Initialized, Role: FOLLOWER
I20260619 00:57:56.809458 20741 ts_tablet_manager.cc:1428] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 00:57:56.809705 20743 consensus_queue.cc:260] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:56.812983 20592 heartbeater.cc:503] Master 127.0.97.62:46065 was elected leader, sending a full tablet report...
I20260619 00:57:56.813434 20743 ts_tablet_manager.cc:1428] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260619 00:57:56.814989 20740 ts_tablet_manager.cc:1428] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Time spent starting tablet: real 0.021s user 0.014s sys 0.001s
W20260619 00:57:56.935784 20723 tablet.cc:2369] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:56.994625 20593 tablet.cc:2369] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:57:57.029886 20462 tablet.cc:2369] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:57:57.108464 20746 raft_consensus.cc:482] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:57.108844 20746 raft_consensus.cc:504] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:57.110639 20746 leader_election.cc:288] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e7fca2103442dd9cd65b1c9ea44d0d (127.0.97.1:38545), 513ebe26dee44defae8963cc6b856b36 (127.0.97.2:43523)
I20260619 00:57:57.114488 20747 raft_consensus.cc:482] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:57:57.114795 20747 raft_consensus.cc:504] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:57.116695 20747 leader_election.cc:288] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80e7a371a8814905b8d458ae17231926 (127.0.97.3:39539), 513ebe26dee44defae8963cc6b856b36 (127.0.97.2:43523)
I20260619 00:57:57.118335 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.111131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51679 (local address 127.0.97.1:38545)
0619 00:57:57.111347 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.111355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.111602 (+ 247us) server_negotiation.cc:407] Connection header received
0619 00:57:57.111764 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.111769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.111871 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.111965 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:57.112458 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.113314 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.115791 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.117276 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.117425 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.117514 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.117870 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.118016 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:57:57.118947 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.111041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.97.3:51679)
0619 00:57:57.111395 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.111417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:57.111584 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:57.112027 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:57.112034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:57.112070 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:57.112306 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.112319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.113459 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.113466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.115635 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.115650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.117432 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.117438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.117645 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.117695 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 00:57:57.118471 (+ 776us) client_negotiation.cc:240] Negotiation successful
0619 00:57:57.118655 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260619 00:57:57.119639 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.111304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52349 (local address 127.0.97.2:43523)
0619 00:57:57.111553 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.111559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.112109 (+ 550us) server_negotiation.cc:407] Connection header received
0619 00:57:57.112347 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.112351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.112408 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.112508 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:57.113120 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.114093 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.116604 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.118080 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.118214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.118326 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.119076 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.119293 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:57:57.120461 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.111188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43523 (local address 127.0.97.3:52349)
0619 00:57:57.112010 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.112027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:57.112210 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:57.112530 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:57.112534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:57.112594 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:57.112961 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.112970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.114303 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.114313 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.116395 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.116409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.118451 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.118459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.118771 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.118839 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 00:57:57.119956 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0619 00:57:57.120115 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":422,"threads_started":1}
I20260619 00:57:57.120407 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "80e7a371a8814905b8d458ae17231926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" is_pre_election: true
I20260619 00:57:57.121181 20416 raft_consensus.cc:2454] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e7a371a8814905b8d458ae17231926 in term 0.
I20260619 00:57:57.121659 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "80e7a371a8814905b8d458ae17231926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513ebe26dee44defae8963cc6b856b36" is_pre_election: true
I20260619 00:57:57.122400 20547 raft_consensus.cc:2454] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e7a371a8814905b8d458ae17231926 in term 0.
I20260619 00:57:57.122781 20613 leader_election.cc:302] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e7fca2103442dd9cd65b1c9ea44d0d, 80e7a371a8814905b8d458ae17231926; no voters:
I20260619 00:57:57.123343 20746 raft_consensus.cc:2781] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:57:57.123525 20746 raft_consensus.cc:482] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:57:57.123667 20746 raft_consensus.cc:3037] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:57.126993 20746 raft_consensus.cc:504] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:57.127820 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.118822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37209 (local address 127.0.97.2:43523)
0619 00:57:57.119886 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.119893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.119913 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:57:57.120130 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.120139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.120261 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.120336 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:57.121410 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.122945 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.125456 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.126604 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.126733 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.126846 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.127371 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.127510 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":4922,"threads_started":1}
I20260619 00:57:57.127962 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.118209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43523 (local address 127.0.97.1:37209)
0619 00:57:57.119802 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.119821 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:57.119936 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:57.120556 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:57.120563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:57.120617 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:57.121141 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.121158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.123175 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.123184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.125276 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.125288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.126842 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.126850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.127085 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.127152 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 00:57:57.127265 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 00:57:57.127407 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"mutex_wait_us":24,"thread_start_us":1190,"threads_started":1}
I20260619 00:57:57.128235 20416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "80e7a371a8814905b8d458ae17231926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d"
I20260619 00:57:57.128449 20746 leader_election.cc:288] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [CANDIDATE]: Term 1 election: Requested vote from peers 77e7fca2103442dd9cd65b1c9ea44d0d (127.0.97.1:38545), 513ebe26dee44defae8963cc6b856b36 (127.0.97.2:43523)
I20260619 00:57:57.128475 20416 raft_consensus.cc:3037] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:57.128571 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "80e7a371a8814905b8d458ae17231926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513ebe26dee44defae8963cc6b856b36"
I20260619 00:57:57.128851 20547 raft_consensus.cc:3037] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:57.129884 20546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513ebe26dee44defae8963cc6b856b36" is_pre_election: true
I20260619 00:57:57.131731 20416 raft_consensus.cc:2454] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e7a371a8814905b8d458ae17231926 in term 1.
I20260619 00:57:57.131968 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.117030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55099 (local address 127.0.97.3:39539)
0619 00:57:57.117212 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.117217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.118119 (+ 902us) server_negotiation.cc:407] Connection header received
0619 00:57:57.118387 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.118392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.118469 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.118533 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:57:57.124640 (+ 6107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.125865 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.129626 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.130806 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.130927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.131025 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.131565 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.131713 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:57:57.132071 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.117147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39539 (local address 127.0.97.1:55099)
0619 00:57:57.118021 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.118038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:57:57.118225 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:57:57.123956 (+ 5731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:57:57.123962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:57:57.123995 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:57:57.124460 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.124472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.126027 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.126033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.129393 (+ 3360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:57:57.129411 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.131189 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.131195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:57:57.131360 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.131389 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:57:57.131526 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 00:57:57.131684 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":3,"thread_start_us":542,"threads_started":1}
I20260619 00:57:57.132155 20613 leader_election.cc:302] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e7fca2103442dd9cd65b1c9ea44d0d, 80e7a371a8814905b8d458ae17231926; no voters:
I20260619 00:57:57.132344 20547 raft_consensus.cc:2454] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e7a371a8814905b8d458ae17231926 in term 1.
I20260619 00:57:57.132443 20746 raft_consensus.cc:2781] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:57.132977 20746 raft_consensus.cc:686] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 1 LEADER]: Becoming Leader. State: Replica: 80e7a371a8814905b8d458ae17231926, State: Running, Role: LEADER
I20260619 00:57:57.133308 20677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4765d1020a4a0f8314f3a0f4abd5ec" candidate_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e7a371a8814905b8d458ae17231926" is_pre_election: true
I20260619 00:57:57.133797 20746 consensus_queue.cc:237] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } }
I20260619 00:57:57.134334 20349 leader_election.cc:302] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e7fca2103442dd9cd65b1c9ea44d0d; no voters: 513ebe26dee44defae8963cc6b856b36, 80e7a371a8814905b8d458ae17231926
I20260619 00:57:57.135033 20747 raft_consensus.cc:2726] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 00:57:57.138738 20274 catalog_manager.cc:5495] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80e7a371a8814905b8d458ae17231926 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "80e7a371a8814905b8d458ae17231926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e7a371a8814905b8d458ae17231926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 } health_report { overall_health: UNKNOWN } } }
I20260619 00:57:57.215929 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.210661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.0.97.3:39539)
0619 00:57:57.210858 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.210873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.210890 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:57:57.211113 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.211117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.211210 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.211365 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:57:57.211654 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.212501 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.213941 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.214320 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.214411 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.214865 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:57:57.215363 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:57:57.215468 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.215531 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.215588 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 00:57:57.223961 20547 raft_consensus.cc:1261] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 1 FOLLOWER]: Refusing update from remote peer 80e7a371a8814905b8d458ae17231926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:57.224004 20416 raft_consensus.cc:1261] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 1 FOLLOWER]: Refusing update from remote peer 80e7a371a8814905b8d458ae17231926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:57:57.224983 20746 consensus_queue.cc:1035] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513ebe26dee44defae8963cc6b856b36" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:57.225296 20753 consensus_queue.cc:1035] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e7fca2103442dd9cd65b1c9ea44d0d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:57:57.253962 20756 mvcc.cc:205] Tried to move back new op lower bound from 7298378453900161024 to 7298378453545029632. Current Snapshot: MvccSnapshot[applied={T|T < 7298378453900161024}]
I20260619 00:57:57.254673 20758 mvcc.cc:205] Tried to move back new op lower bound from 7298378453900161024 to 7298378453545029632. Current Snapshot: MvccSnapshot[applied={T|T < 7298378453900161024}]
I20260619 00:57:57.277493 20274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:57.277777 20274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:57:57.281239 20274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 743da4bb59554a428189dff6f26a85d7: Sending DeleteTablet for 3 replicas of tablet 2c4765d1020a4a0f8314f3a0f4abd5ec
I20260619 00:57:57.282516 20396 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:57 UTC) from {username='slave'} at 127.0.0.1:55540
I20260619 00:57:57.282557 20657 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:57 UTC) from {username='slave'} at 127.0.0.1:55408
I20260619 00:57:57.282897 20527 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:57:57 UTC) from {username='slave'} at 127.0.0.1:35590
I20260619 00:57:57.284286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20334
I20260619 00:57:57.298472 20767 tablet_replica.cc:330] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: stopping tablet replica
I20260619 00:57:57.299165 20767 raft_consensus.cc:2229] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:57:57.300101 20767 raft_consensus.cc:2258] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:57.301066 20768 tablet_replica.cc:330] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: stopping tablet replica
I20260619 00:57:57.301568 20768 raft_consensus.cc:2229] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:57.302112 20766 tablet_replica.cc:330] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d: stopping tablet replica
I20260619 00:57:57.302165 20768 raft_consensus.cc:2258] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:57.302561 20766 raft_consensus.cc:2229] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:57:57.303108 20766 raft_consensus.cc:2258] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 77e7fca2103442dd9cd65b1c9ea44d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:57:57.304404 20768 ts_tablet_manager.cc:1905] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:57.304736 20767 ts_tablet_manager.cc:1905] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:57:57.310815 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.285619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.97.1:38545)
0619 00:57:57.285817 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.285823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.301480 (+ 15657us) server_negotiation.cc:407] Connection header received
0619 00:57:57.301722 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.301728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.301854 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.301965 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:57.302956 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.304396 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.306604 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.307285 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.307391 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.308997 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:57.309010 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:57.309027 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:57.309850 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:57.309910 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:57.309935 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:57.310029 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.310313 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.310438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:57:57.311981 20768 ts_tablet_manager.cc:1918] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:57.312156 20768 log.cc:1196] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-1/wal/wals/2c4765d1020a4a0f8314f3a0f4abd5ec
I20260619 00:57:57.312387 20767 ts_tablet_manager.cc:1918] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:57:57.312479 20456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:57.312568 20767 log.cc:1196] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Deleting WAL directory at /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830351410501-388-0/minicluster-data/ts-2/wal/wals/2c4765d1020a4a0f8314f3a0f4abd5ec
I20260619 00:57:57.312772 20768 ts_tablet_manager.cc:1939] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 513ebe26dee44defae8963cc6b856b36: Deleting consensus metadata
I20260619 00:57:57.313095 20767 ts_tablet_manager.cc:1939] T 2c4765d1020a4a0f8314f3a0f4abd5ec P 80e7a371a8814905b8d458ae17231926: Deleting consensus metadata
I20260619 00:57:57.314767 20259 catalog_manager.cc:4841] TS 513ebe26dee44defae8963cc6b856b36 (127.0.97.2:43523): tablet 2c4765d1020a4a0f8314f3a0f4abd5ec (table test-table [id=ab37b4e1b62b4eb58351bc0f08b73c9d]) successfully deleted
I20260619 00:57:57.315241 20259 catalog_manager.cc:4841] TS 80e7a371a8814905b8d458ae17231926 (127.0.97.3:39539): tablet 2c4765d1020a4a0f8314f3a0f4abd5ec (table test-table [id=ab37b4e1b62b4eb58351bc0f08b73c9d]) successfully deleted
I20260619 00:57:57.408434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20334
I20260619 00:57:57.432356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20465
W20260619 00:57:57.432700 20262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38545 (error 108)
I20260619 00:57:57.435106 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.434367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55556)
0619 00:57:57.434827 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.434929 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
W20260619 00:57:57.435395 20262 catalog_manager.cc:4584] TS 77e7fca2103442dd9cd65b1c9ea44d0d (127.0.97.1:38545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4765d1020a4a0f8314f3a0f4abd5ec: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
I20260619 00:57:57.440533 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.433266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.97.2:43523)
0619 00:57:57.433508 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.433517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.433544 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:57:57.433770 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.433778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.433887 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.433982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:57.434611 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.435721 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.437168 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.437858 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.437987 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.438971 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:57.438989 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:57.439006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:57.439803 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:57.439866 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:57.439890 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:57.440003 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.440096 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.440196 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 00:57:57.441242 20587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:57.489137 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.488522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55570)
0619 00:57:57.488804 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.488942 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 00:57:57.490477 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.489995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55586)
0619 00:57:57.490167 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.490305 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:57:57.536113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20465
I20260619 00:57:57.551424 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.550775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55600)
0619 00:57:57.551077 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.551213 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 00:57:57.552858 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.552332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55610)
0619 00:57:57.552524 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.552631 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 00:57:57.553715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20595
I20260619 00:57:57.560992 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.554434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.0.97.3:39539)
0619 00:57:57.554608 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:57.554615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:57.554685 (+ 70us) server_negotiation.cc:407] Connection header received
0619 00:57:57.554907 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:57.554912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:57.555057 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:57.555179 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:57.555574 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.556368 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.557727 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:57.558206 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:57.558347 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:57.559492 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:57.559503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:57.559521 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:57:57.560132 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:57.560209 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:57:57.560230 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:57.560345 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:57.560482 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 00:57:57.560600 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:57:57.561774 20717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:57.633759 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.633267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55614)
0619 00:57:57.633501 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.633599 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 00:57:57.634892 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.634513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55626)
0619 00:57:57.634696 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.634762 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 00:57:57.686408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20595
I20260619 00:57:57.706317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20244
I20260619 00:57:57.707422 20305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:57.818331 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:57.817297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38545 (local address 127.0.0.1:55636)
0619 00:57:57.818041 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 00:57:57.818179 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573}
I20260619 00:57:57.818398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20244
2026-06-19T00:57:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3116 ms)
[----------] 6 tests from EncryptionPolicyTest (22384 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:57:57 dist-test-slave-r7bh krb5kdc[20778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:57:57 dist-test-slave-r7bh krb5kdc[20778](info): set up 2 sockets
Jun 19 00:57:57 dist-test-slave-r7bh krb5kdc[20778](info): commencing operation
krb5kdc: starting...
W20260619 00:57:58.941319 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:57:59.196221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:57:59.537366 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:57:59.537705 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:57:59.537812 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:57:59.545117 20791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:57:59.545250 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:57:59.545305 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:57:59.545367 20791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:57:59.545411 20791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:57:59.555492 20791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:57:59.556771 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:57:59.558058 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:57:59.570289 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:59.570257 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:57:59.570346 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:57:59.571130 20791 server_base.cc:1034] running on GCE node
I20260619 00:57:59.571904 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:57:59.573388 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:57:59.574599 20791 hybrid_clock.cc:648] HybridClock initialized: now 1781830679574552 us; error 38 us; skew 500 ppm
I20260619 00:57:59.574954 20791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:57:59.581247 20791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:57:59.582731 20791 webserver.cc:460] Webserver started at http://127.0.0.1:39369/ using document root <none> and password file <none>
I20260619 00:57:59.583374 20791 fs_manager.cc:359] Metadata directory not provided
I20260619 00:57:59.583482 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:57:59.583742 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:57:59.586933 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "7bcb894621dd412ab01a2a152e0058ee"
format_stamp: "Formatted at 2026-06-19 00:57:59 on dist-test-slave-r7bh"
server_key: "001e3120a2313ef677901c2b6f5de0de"
server_key_iv: "1b9c302d6ca2a84d367451011183638b"
server_key_version: "encryptionkey@0"
I20260619 00:57:59.587787 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "7bcb894621dd412ab01a2a152e0058ee"
format_stamp: "Formatted at 2026-06-19 00:57:59 on dist-test-slave-r7bh"
server_key: "001e3120a2313ef677901c2b6f5de0de"
server_key_iv: "1b9c302d6ca2a84d367451011183638b"
server_key_version: "encryptionkey@0"
I20260619 00:57:59.593366 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 00:57:59.597177 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:57:59.598733 20791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:57:59.598948 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "7bcb894621dd412ab01a2a152e0058ee"
format_stamp: "Formatted at 2026-06-19 00:57:59 on dist-test-slave-r7bh"
server_key: "001e3120a2313ef677901c2b6f5de0de"
server_key_iv: "1b9c302d6ca2a84d367451011183638b"
server_key_version: "encryptionkey@0"
I20260619 00:57:59.599179 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:57:59.617957 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:57:59.622232 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:57:59.622525 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:57:59.640868 20791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46185
I20260619 00:57:59.640933 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46185 every 8 connection(s)
I20260619 00:57:59.642277 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:57:59.646001 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:57:59.651535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20791
I20260619 00:57:59.651749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:57:59.652367 388 external_mini_cluster.cc:1428] Setting key 2a341b0a881b14dc5dba36014577caf4
I20260619 00:57:59.656880 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee: Bootstrap starting.
I20260619 00:57:59.660676 20858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee: Neither blocks nor log segments found. Creating new log.
I20260619 00:57:59.661911 20858 log.cc:824] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee: Log is configured to *not* fsync() on all Append() calls
I20260619 00:57:59.665331 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee: No bootstrap required, opened a new log
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 00:57:59.673874 20858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } }
I20260619 00:57:59.674296 20858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:57:59.674407 20858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcb894621dd412ab01a2a152e0058ee, State: Initialized, Role: FOLLOWER
I20260619 00:57:59.674988 20858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } }
I20260619 00:57:59.675253 20858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:57:59.675439 20858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:57:59.675649 20858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:57:59.677469 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:59.654510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53227 (local address 127.0.0.1:46185)
0619 00:57:59.655380 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:59.655392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:59.655436 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:57:59.657326 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:59.657350 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:59.657769 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:59.658116 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:59.660019 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:59.661472 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:59.663189 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:59.663873 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:59.664013 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:59.669149 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:59.669157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:59.669179 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:59.669224 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:59.672490 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:59.673399 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:59.673407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:59.673414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:59.673592 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:59.674048 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:59.674053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:59.674055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:59.674781 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:59.675125 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:59.675633 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 00:57:59.676008 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":437,"threads_started":1}
I20260619 00:57:59.678690 20858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } }
I20260619 00:57:59.679256 20858 leader_election.cc:302] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bcb894621dd412ab01a2a152e0058ee; no voters:
I20260619 00:57:59.679850 20858 leader_election.cc:288] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:57:59.679973 20863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:57:59.681234 20863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [term 1 LEADER]: Becoming Leader. State: Replica: 7bcb894621dd412ab01a2a152e0058ee, State: Running, Role: LEADER
I20260619 00:57:59.681787 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:57:59.681990 20863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } }
I20260619 00:57:59.682232 20858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:57:59.690734 20865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bcb894621dd412ab01a2a152e0058ee. Latest consensus state: current_term: 1 leader_uuid: "7bcb894621dd412ab01a2a152e0058ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } } }
I20260619 00:57:59.691210 20865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:59.691857 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bcb894621dd412ab01a2a152e0058ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bcb894621dd412ab01a2a152e0058ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46185 } } }
I20260619 00:57:59.692319 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee [sys.catalog]: This master's current role is: LEADER
I20260619 00:57:59.696854 20876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:57:59.704056 20876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:57:59.717000 20876 catalog_manager.cc:1347] Generated new cluster ID: 9790995f32904638afae130b65f0eaa8
I20260619 00:57:59.717126 20876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:57:59.730213 20876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:57:59.732033 20876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:57:59.734807 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:57:59.699892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60093 (local address 127.0.0.1:46185)
0619 00:57:59.700075 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 00:57:59.700082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:57:59.717645 (+ 17563us) server_negotiation.cc:407] Connection header received
0619 00:57:59.717943 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:57:59.717948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:57:59.718047 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:57:59.718131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:57:59.720335 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:59.721537 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:59.723397 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:57:59.724060 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:57:59.724173 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:57:59.727634 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:57:59.727645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:57:59.727649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:57:59.727692 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:57:59.731511 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:59.732457 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:59.732463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:59.732466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:59.732625 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:57:59.733160 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:57:59.733167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:57:59.733170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:57:59.733585 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:57:59.733868 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 00:57:59.734233 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 00:57:59.734397 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":67}
I20260619 00:57:59.743358 20876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bcb894621dd412ab01a2a152e0058ee: Generated new TSK 0
I20260619 00:57:59.744210 20876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:57:59.753057 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20791
I20260619 00:57:59.754287 20853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:57:59.850051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20791
W20260619 00:57:59.867314 20776 connection.cc:535] client connection to 127.0.0.1:46185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46185): Connection reset by peer (error 104)
2026-06-19T00:57:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20891](info): set up 2 sockets
Jun 19 00:57:59 dist-test-slave-r7bh krb5kdc[20891](info): commencing operation
krb5kdc: starting...
W20260619 00:58:00.957237 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:01 dist-test-slave-r7bh krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:01.219694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:01.558208 20904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:01.558524 20904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:01.558696 20904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:01.566442 20904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:01.566560 20904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:01.566663 20904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:01.566766 20904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:01.566843 20904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:01.576854 20904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:01.578203 20904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:01.579553 20904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:01.591238 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:01.591200 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:01.591269 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:01.592249 20904 server_base.cc:1034] running on GCE node
I20260619 00:58:01.593093 20904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:01.594457 20904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:01.595662 20904 hybrid_clock.cc:648] HybridClock initialized: now 1781830681595617 us; error 42 us; skew 500 ppm
I20260619 00:58:01.595995 20904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:01 dist-test-slave-r7bh krb5kdc[20891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:01.599103 20904 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:01.600462 20904 webserver.cc:460] Webserver started at http://127.0.0.1:39103/ using document root <none> and password file <none>
I20260619 00:58:01.601155 20904 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:01.601256 20904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:01.601511 20904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:01.604454 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "8b7b718f1827466cbda7616b26efd338"
format_stamp: "Formatted at 2026-06-19 00:58:01 on dist-test-slave-r7bh"
server_key: "dd7c2ae25eca616a5e7d250c11fd7903"
server_key_iv: "991688894b36b23b8f9cb904f930f6af"
server_key_version: "encryptionkey@0"
I20260619 00:58:01.605232 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b7b718f1827466cbda7616b26efd338"
format_stamp: "Formatted at 2026-06-19 00:58:01 on dist-test-slave-r7bh"
server_key: "dd7c2ae25eca616a5e7d250c11fd7903"
server_key_iv: "991688894b36b23b8f9cb904f930f6af"
server_key_version: "encryptionkey@0"
I20260619 00:58:01.610244 20904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 00:58:01.613528 20919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:01.614780 20904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:01.614955 20904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "8b7b718f1827466cbda7616b26efd338"
format_stamp: "Formatted at 2026-06-19 00:58:01 on dist-test-slave-r7bh"
server_key: "dd7c2ae25eca616a5e7d250c11fd7903"
server_key_iv: "991688894b36b23b8f9cb904f930f6af"
server_key_version: "encryptionkey@0"
I20260619 00:58:01.615154 20904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:01.638872 20904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:01.643386 20904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:01.643687 20904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:01.661675 20904 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44943
I20260619 00:58:01.661710 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44943 every 8 connection(s)
I20260619 00:58:01.663040 20904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:01.664093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 20904
I20260619 00:58:01.664269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:01.664703 388 external_mini_cluster.cc:1428] Setting key f75600c874e04b4074570f263bd75329
I20260619 00:58:01.667586 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:01.679936 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:01.666514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47821 (local address 127.0.0.1:44943)
0619 00:58:01.667277 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:01.667292 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:01.667342 (+ 50us) server_negotiation.cc:407] Connection header received
0619 00:58:01.669777 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:01.669806 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:01.670289 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:01.670773 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:01.671293 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:01.673329 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:01.674643 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:01.675648 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:01.675851 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:01.677075 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:01.677086 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:01.677105 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:01.677877 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:01.677971 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:01.678003 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:01.678184 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:01.678422 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 00:58:01.678617 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":314,"threads_started":1}
I20260619 00:58:01.681569 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338: Bootstrap starting.
I20260619 00:58:01.683688 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:01.684922 20971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:01.685954 20971 log.cc:824] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:01.688957 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338: No bootstrap required, opened a new log
I20260619 00:58:01.694864 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:01.688269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47903 (local address 127.0.0.1:44943)
0619 00:58:01.688439 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:01.688446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:01.688761 (+ 315us) server_negotiation.cc:407] Connection header received
0619 00:58:01.688971 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:01.688979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:01.689060 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:01.689164 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:01.689675 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:01.690801 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:01.692089 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:01.692795 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:01.692948 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:01.694046 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:01.694053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:01.694058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:01.694079 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:01.694151 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:01.694176 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:01.694301 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:01.694462 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 00:58:01.694519 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:58:01.697872 20971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260619 00:58:01.698210 20971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:01.698287 20971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7b718f1827466cbda7616b26efd338, State: Initialized, Role: FOLLOWER
I20260619 00:58:01.698758 20971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260619 00:58:01.698946 20971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:01.699064 20971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:01.699198 20971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:01.701313 20971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260619 00:58:01.701786 20971 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7b718f1827466cbda7616b26efd338; no voters:
I20260619 00:58:01.702368 20971 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:01.702525 20981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:01.703210 20981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7b718f1827466cbda7616b26efd338, State: Running, Role: LEADER
I20260619 00:58:01.703722 20981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } }
I20260619 00:58:01.704361 20971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:01.706643 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7b718f1827466cbda7616b26efd338. Latest consensus state: current_term: 1 leader_uuid: "8b7b718f1827466cbda7616b26efd338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } } }
I20260619 00:58:01.706655 20982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7b718f1827466cbda7616b26efd338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b718f1827466cbda7616b26efd338" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44943 } } }
I20260619 00:58:01.707657 20982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:01.707036 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:01.712975 20987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:01.717226 20987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:01.725523 20987 catalog_manager.cc:1347] Generated new cluster ID: 23e106b594b840b1ba8acf69cec4cb66
I20260619 00:58:01.725641 20987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:01.759613 20987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:01.761015 20987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:01.774046 20987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7b718f1827466cbda7616b26efd338: Generated new TSK 0
I20260619 00:58:01.774935 20987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:01.783169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20904
W20260619 00:58:01.784276 20966 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47821
W20260619 00:58:01.785177 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 20904: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:01.785383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 20904
W20260619 00:58:01.799798 20889 connection.cc:535] client connection to 127.0.0.1:44943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44943): Connection reset by peer (error 104)
2026-06-19T00:58:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:01 dist-test-slave-r7bh krb5kdc[21002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:01 dist-test-slave-r7bh krb5kdc[21002](info): set up 2 sockets
Jun 19 00:58:01 dist-test-slave-r7bh krb5kdc[21002](info): commencing operation
krb5kdc: starting...
W20260619 00:58:02.885241 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:03 dist-test-slave-r7bh krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:03.140034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:03.485199 21015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:03.485519 21015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:03.485692 21015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:03.493095 21015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:03.493211 21015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:03.493333 21015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:03.493455 21015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:03.493538 21015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:03.503536 21015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:03.504951 21015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:03.506290 21015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:03.517868 21020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:03.517930 21021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:03.517953 21023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:03.519165 21015 server_base.cc:1034] running on GCE node
I20260619 00:58:03.520080 21015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:03.521679 21015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:03.522946 21015 hybrid_clock.cc:648] HybridClock initialized: now 1781830683522910 us; error 49 us; skew 500 ppm
I20260619 00:58:03.523332 21015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:03 dist-test-slave-r7bh krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:03.527047 21015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:03.528729 21015 webserver.cc:460] Webserver started at http://127.0.0.1:34367/ using document root <none> and password file <none>
I20260619 00:58:03.529502 21015 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:03.529659 21015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:03.529986 21015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:03.533255 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "67c9dab06fd5435782622b1aa9f3954a"
format_stamp: "Formatted at 2026-06-19 00:58:03 on dist-test-slave-r7bh"
server_key: "6389ff50c6f75c9efe82ff789eed4a7f"
server_key_iv: "7c1eebeadab643c92698aa1fa9845bfd"
server_key_version: "encryptionkey@0"
I20260619 00:58:03.534075 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "67c9dab06fd5435782622b1aa9f3954a"
format_stamp: "Formatted at 2026-06-19 00:58:03 on dist-test-slave-r7bh"
server_key: "6389ff50c6f75c9efe82ff789eed4a7f"
server_key_iv: "7c1eebeadab643c92698aa1fa9845bfd"
server_key_version: "encryptionkey@0"
I20260619 00:58:03.539518 21015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:58:03.543114 21030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:03.544597 21015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:58:03.544791 21015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "67c9dab06fd5435782622b1aa9f3954a"
format_stamp: "Formatted at 2026-06-19 00:58:03 on dist-test-slave-r7bh"
server_key: "6389ff50c6f75c9efe82ff789eed4a7f"
server_key_iv: "7c1eebeadab643c92698aa1fa9845bfd"
server_key_version: "encryptionkey@0"
I20260619 00:58:03.545089 21015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:03.551093 21015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:03.555557 21015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:03.555876 21015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:03.575764 21015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37833
I20260619 00:58:03.575829 21081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37833 every 8 connection(s)
I20260619 00:58:03.577303 21015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:03.581101 21082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:03.585115 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21015
I20260619 00:58:03.585309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:03.585742 388 external_mini_cluster.cc:1428] Setting key 49a3d57aecdd76b4d4a8d552b4c76055
I20260619 00:58:03.593299 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: Bootstrap starting.
I20260619 00:58:03.595805 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:03.587766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56557 (local address 127.0.0.1:37833)
0619 00:58:03.588748 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:03.588774 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:03.589191 (+ 417us) server_negotiation.cc:407] Connection header received
0619 00:58:03.591810 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:03.591840 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:03.592293 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:03.592738 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:03.593805 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:03.593813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:03.593830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:03.593855 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:03.593934 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:03.593962 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:03.594176 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:03.594308 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 00:58:03.594494 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":406,"threads_started":1}
I20260619 00:58:03.597136 21082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:03.598368 21082 log.cc:824] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:03.599684 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:03.601300 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: No bootstrap required, opened a new log
I20260619 00:58:03.606550 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:03.603980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43325 (local address 127.0.0.1:37833)
0619 00:58:03.604169 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:03.604176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:03.604581 (+ 405us) server_negotiation.cc:407] Connection header received
0619 00:58:03.604726 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:03.604732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:03.604790 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:03.604941 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:03.605819 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:03.605824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:03.605829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:03.605846 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:03.605896 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:03.605918 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:03.606068 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:03.606193 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 00:58:03.606259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 00:58:03.609874 21082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260619 00:58:03.610247 21082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:03.610321 21082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c9dab06fd5435782622b1aa9f3954a, State: Initialized, Role: FOLLOWER
I20260619 00:58:03.610906 21082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260619 00:58:03.611181 21082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:03.611327 21082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:03.611521 21082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:03.614020 21082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260619 00:58:03.614599 21082 leader_election.cc:302] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67c9dab06fd5435782622b1aa9f3954a; no voters:
I20260619 00:58:03.615252 21082 leader_election.cc:288] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:03.615437 21092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:03.616925 21092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [term 1 LEADER]: Becoming Leader. State: Replica: 67c9dab06fd5435782622b1aa9f3954a, State: Running, Role: LEADER
I20260619 00:58:03.617426 21092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } }
I20260619 00:58:03.617684 21082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:03.625656 21094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c9dab06fd5435782622b1aa9f3954a. Latest consensus state: current_term: 1 leader_uuid: "67c9dab06fd5435782622b1aa9f3954a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } } }
I20260619 00:58:03.626284 21094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:03.625818 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67c9dab06fd5435782622b1aa9f3954a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c9dab06fd5435782622b1aa9f3954a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37833 } } }
I20260619 00:58:03.626765 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a [sys.catalog]: This master's current role is: LEADER
W20260619 00:58:03.631129 21106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:03.631335 21106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:03.631539 21107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:03.633946 21107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:03.643060 21107 catalog_manager.cc:1347] Generated new cluster ID: e86365b89e714adfaa9c7a24296728d2
I20260619 00:58:03.643183 21107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:03.661454 21107 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:03.662004 21107 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:03.671610 21107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c9dab06fd5435782622b1aa9f3954a: Generated new TSK 0
I20260619 00:58:03.672766 21107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:03.676802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21015
W20260619 00:58:03.678469 21077 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56557
W20260619 00:58:03.679188 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21015: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:03.679371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21015
2026-06-19T00:58:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:03 dist-test-slave-r7bh krb5kdc[21113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:03 dist-test-slave-r7bh krb5kdc[21113](info): set up 2 sockets
Jun 19 00:58:03 dist-test-slave-r7bh krb5kdc[21113](info): commencing operation
krb5kdc: starting...
W20260619 00:58:04.778000 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:04 dist-test-slave-r7bh krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:05.037827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:05.370131 21126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:05.370447 21126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:05.370594 21126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:05.378080 21126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:05.378253 21126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:05.378317 21126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:05.378396 21126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:05.378446 21126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:05.388592 21126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:05.390075 21126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:05.391574 21126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:05.405632 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:05.406572 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:05.406601 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:05.406816 21126 server_base.cc:1034] running on GCE node
I20260619 00:58:05.407729 21126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:05.409435 21126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:05.410681 21126 hybrid_clock.cc:648] HybridClock initialized: now 1781830685410624 us; error 39 us; skew 500 ppm
I20260619 00:58:05.411079 21126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:05 dist-test-slave-r7bh krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:05.418313 21126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:05.420477 21126 webserver.cc:460] Webserver started at http://127.0.0.1:40019/ using document root <none> and password file <none>
I20260619 00:58:05.421569 21126 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:05.421751 21126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:05.422158 21126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:05.425213 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "4640a9c9f00647cf921d47366f060039"
format_stamp: "Formatted at 2026-06-19 00:58:05 on dist-test-slave-r7bh"
server_key: "5f5f53df636aa88b7934948e5281e2a8"
server_key_iv: "d1d487adf320c2386b59a8358fa9a850"
server_key_version: "encryptionkey@0"
I20260619 00:58:05.425940 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "4640a9c9f00647cf921d47366f060039"
format_stamp: "Formatted at 2026-06-19 00:58:05 on dist-test-slave-r7bh"
server_key: "5f5f53df636aa88b7934948e5281e2a8"
server_key_iv: "d1d487adf320c2386b59a8358fa9a850"
server_key_version: "encryptionkey@0"
I20260619 00:58:05.431115 21126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:58:05.434913 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:05.436496 21126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:58:05.436650 21126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "4640a9c9f00647cf921d47366f060039"
format_stamp: "Formatted at 2026-06-19 00:58:05 on dist-test-slave-r7bh"
server_key: "5f5f53df636aa88b7934948e5281e2a8"
server_key_iv: "d1d487adf320c2386b59a8358fa9a850"
server_key_version: "encryptionkey@0"
I20260619 00:58:05.436872 21126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:05.457692 21126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:05.462322 21126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:05.462574 21126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:05.482869 21126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38515
I20260619 00:58:05.482903 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38515 every 8 connection(s)
I20260619 00:58:05.484459 21126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:05.488336 21193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:05.493199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21126
I20260619 00:58:05.493402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:05.493835 388 external_mini_cluster.cc:1428] Setting key 757579f5494082a1531ebea478abc882
I20260619 00:58:05.499657 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: Bootstrap starting.
I20260619 00:58:05.503466 21193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:05.504737 21193 log.cc:824] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:05.507654 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: No bootstrap required, opened a new log
Jun 19 00:58:05 dist-test-slave-r7bh krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 00:58:05.518560 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:05.495591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58405 (local address 127.0.0.1:38515)
0619 00:58:05.496410 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:05.496422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:05.496463 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:58:05.498306 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:05.498329 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:05.498744 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:05.499079 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:05.500923 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:05.502249 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:05.504176 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:05.505055 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:05.505206 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:05.509561 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:05.509572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:05.509596 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:05.509647 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:05.513790 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:05.514517 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:05.514531 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:05.514541 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:05.514749 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:05.515100 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:05.515105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:05.515111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:05.515938 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:05.516249 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:05.516608 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 00:58:05.517142 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":374,"threads_started":1}
I20260619 00:58:05.518939 21193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260619 00:58:05.519452 21193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:05.519574 21193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4640a9c9f00647cf921d47366f060039, State: Initialized, Role: FOLLOWER
I20260619 00:58:05.520200 21193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260619 00:58:05.520478 21193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:05.520637 21193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:05.520896 21193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:05.522821 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:05.523631 21193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260619 00:58:05.524144 21193 leader_election.cc:302] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4640a9c9f00647cf921d47366f060039; no voters:
I20260619 00:58:05.524761 21193 leader_election.cc:288] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:05.524963 21198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:05.526923 21193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:05.533895 21198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [term 1 LEADER]: Becoming Leader. State: Replica: 4640a9c9f00647cf921d47366f060039, State: Running, Role: LEADER
I20260619 00:58:05.534940 21198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } }
I20260619 00:58:05.539822 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4640a9c9f00647cf921d47366f060039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } } }
I20260619 00:58:05.540247 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:05.540803 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4640a9c9f00647cf921d47366f060039. Latest consensus state: current_term: 1 leader_uuid: "4640a9c9f00647cf921d47366f060039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4640a9c9f00647cf921d47366f060039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38515 } } }
I20260619 00:58:05.541113 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039 [sys.catalog]: This master's current role is: LEADER
W20260619 00:58:05.550118 21217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:05.550269 21217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:05.550438 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:05.552428 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:05.556887 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:05.537198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46157 (local address 127.0.0.1:38515)
0619 00:58:05.537445 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:05.537454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:05.537997 (+ 543us) server_negotiation.cc:407] Connection header received
0619 00:58:05.538288 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:05.538293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:05.538380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:05.538443 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:05.540837 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:05.542225 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:05.544801 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:05.545581 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:05.545715 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:05.549088 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:05.549095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:05.549100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:05.549145 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:05.553321 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:05.554287 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:05.554293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:05.554299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:05.554476 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:05.554977 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:05.554982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:05.554985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:05.555435 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:05.555694 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:05.556132 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 00:58:05.556415 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 00:58:05.562659 21218 catalog_manager.cc:1347] Generated new cluster ID: 8ec3fe5bd9a74a30b50b1fdb62a89ca7
I20260619 00:58:05.562810 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:05.571722 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:05.573196 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:05.581528 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4640a9c9f00647cf921d47366f060039: Generated new TSK 0
I20260619 00:58:05.582422 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:05.587579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21126
I20260619 00:58:05.588604 21188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:05.678215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21126
2026-06-19T00:58:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:05 dist-test-slave-r7bh krb5kdc[21225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:05 dist-test-slave-r7bh krb5kdc[21225](info): set up 2 sockets
Jun 19 00:58:05 dist-test-slave-r7bh krb5kdc[21225](info): commencing operation
krb5kdc: starting...
W20260619 00:58:06.785167 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:06 dist-test-slave-r7bh krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:07.044689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:07.373288 21238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:07.373633 21238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:07.373811 21238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:07.380986 21238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:07.381155 21238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:07.381255 21238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:07.381368 21238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:07.381457 21238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:07.391100 21238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:07.392449 21238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:07.393857 21238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:07.405733 21243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:07.405755 21246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:07.405701 21244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:07.406438 21238 server_base.cc:1034] running on GCE node
I20260619 00:58:07.407248 21238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:07.408720 21238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:07.409973 21238 hybrid_clock.cc:648] HybridClock initialized: now 1781830687409917 us; error 47 us; skew 500 ppm
I20260619 00:58:07.410342 21238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:07 dist-test-slave-r7bh krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:07.413604 21238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:07.415243 21238 webserver.cc:460] Webserver started at http://127.0.0.1:44061/ using document root <none> and password file <none>
I20260619 00:58:07.415962 21238 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:07.416110 21238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:07.416435 21238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:07.419535 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "556a4ef8db75474399a865cae0d55179"
format_stamp: "Formatted at 2026-06-19 00:58:07 on dist-test-slave-r7bh"
server_key: "3cc8f3034cd52b6e4ed185d0ce51786a"
server_key_iv: "df01e33ea0ad52b6766e53dbde58ac58"
server_key_version: "encryptionkey@0"
I20260619 00:58:07.420337 21238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "556a4ef8db75474399a865cae0d55179"
format_stamp: "Formatted at 2026-06-19 00:58:07 on dist-test-slave-r7bh"
server_key: "3cc8f3034cd52b6e4ed185d0ce51786a"
server_key_iv: "df01e33ea0ad52b6766e53dbde58ac58"
server_key_version: "encryptionkey@0"
I20260619 00:58:07.425906 21238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 00:58:07.430167 21253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:07.432021 21238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 00:58:07.432230 21238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "556a4ef8db75474399a865cae0d55179"
format_stamp: "Formatted at 2026-06-19 00:58:07 on dist-test-slave-r7bh"
server_key: "3cc8f3034cd52b6e4ed185d0ce51786a"
server_key_iv: "df01e33ea0ad52b6766e53dbde58ac58"
server_key_version: "encryptionkey@0"
I20260619 00:58:07.432452 21238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:07.462505 21238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:07.467272 21238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:07.467584 21238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:07.484876 21238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43383
I20260619 00:58:07.484894 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43383 every 8 connection(s)
I20260619 00:58:07.486286 21238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:07.490017 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:07.490268 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21238
I20260619 00:58:07.490442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:07.490864 388 external_mini_cluster.cc:1428] Setting key 16e2d92966ff014464fbaffae47b5240
I20260619 00:58:07.505594 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179: Bootstrap starting.
I20260619 00:58:07.505911 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:07.492739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57129 (local address 127.0.0.1:43383)
0619 00:58:07.493652 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:07.493667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:07.493715 (+ 48us) server_negotiation.cc:407] Connection header received
0619 00:58:07.495372 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:07.495405 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:07.495942 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:07.496439 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:07.497047 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:07.499071 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:07.500444 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:07.501562 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:07.501781 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:07.502930 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:07.502943 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:07.502961 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:07.503743 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:07.503829 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:07.503861 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:07.503976 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:07.504211 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 00:58:07.504505 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":352,"threads_started":1}
I20260619 00:58:07.509449 21305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:07.509935 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:07.510906 21305 log.cc:824] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:07.514237 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179: No bootstrap required, opened a new log
I20260619 00:58:07.521559 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:07.514451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38095 (local address 127.0.0.1:43383)
0619 00:58:07.514694 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:07.514702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:07.515184 (+ 482us) server_negotiation.cc:407] Connection header received
0619 00:58:07.515359 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:07.515362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:07.515421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:07.515496 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:07.516131 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:07.516947 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:07.518917 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:07.519379 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:07.519492 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:07.520697 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:07.520701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:07.520707 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:07.520727 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:07.520766 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:07.520780 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:07.520945 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:07.521100 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 00:58:07.521180 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 00:58:07.524145 21305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 00:58:07.524626 21305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:07.524749 21305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556a4ef8db75474399a865cae0d55179, State: Initialized, Role: FOLLOWER
I20260619 00:58:07.525435 21305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 00:58:07.525686 21305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:07.525839 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:07.526031 21305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:07.528206 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 00:58:07.528733 21305 leader_election.cc:302] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556a4ef8db75474399a865cae0d55179; no voters:
I20260619 00:58:07.529469 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:07.529599 21315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:07.530334 21315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [term 1 LEADER]: Becoming Leader. State: Replica: 556a4ef8db75474399a865cae0d55179, State: Running, Role: LEADER
I20260619 00:58:07.530961 21315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260619 00:58:07.531891 21305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:07.534101 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556a4ef8db75474399a865cae0d55179. Latest consensus state: current_term: 1 leader_uuid: "556a4ef8db75474399a865cae0d55179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260619 00:58:07.534080 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556a4ef8db75474399a865cae0d55179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556a4ef8db75474399a865cae0d55179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260619 00:58:07.534499 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:07.534533 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:07.536430 21320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:07.542120 21320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:07.553100 21320 catalog_manager.cc:1347] Generated new cluster ID: 0599ccc932ed4359b26e81040a80f44f
I20260619 00:58:07.553236 21320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:07.566946 21320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:07.568837 21320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:07.579572 21320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556a4ef8db75474399a865cae0d55179: Generated new TSK 0
I20260619 00:58:07.580729 21320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:07.591744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21238
W20260619 00:58:07.593025 21300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57129
W20260619 00:58:07.593709 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21238: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:07.593869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21238
2026-06-19T00:58:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:07 dist-test-slave-r7bh krb5kdc[21336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:07 dist-test-slave-r7bh krb5kdc[21336](info): set up 2 sockets
Jun 19 00:58:07 dist-test-slave-r7bh krb5kdc[21336](info): commencing operation
krb5kdc: starting...
W20260619 00:58:08.701218 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:08 dist-test-slave-r7bh krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:08.966557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:09.301178 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:09.301486 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:09.301578 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:09.308885 21349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:09.309017 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:09.309072 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:09.309134 21349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:09.309180 21349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:09.319325 21349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:09.320640 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:09.322042 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:09.334288 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:09.334249 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:09.334331 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:09.335474 21349 server_base.cc:1034] running on GCE node
I20260619 00:58:09.336272 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:09.337821 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:09.339051 21349 hybrid_clock.cc:648] HybridClock initialized: now 1781830689339007 us; error 41 us; skew 500 ppm
I20260619 00:58:09.339401 21349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:09 dist-test-slave-r7bh krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:09.342621 21349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:09.344233 21349 webserver.cc:460] Webserver started at http://127.0.0.1:44387/ using document root <none> and password file <none>
I20260619 00:58:09.344992 21349 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:09.345144 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:09.345460 21349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:09.348582 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "d224bfc3134a4e08b644926bb5685607"
format_stamp: "Formatted at 2026-06-19 00:58:09 on dist-test-slave-r7bh"
server_key: "3af4834f007d210755c1597f0d75c5ec"
server_key_iv: "5f12ce5ae24bb5329c331c895749c9fe"
server_key_version: "encryptionkey@0"
I20260619 00:58:09.349442 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "d224bfc3134a4e08b644926bb5685607"
format_stamp: "Formatted at 2026-06-19 00:58:09 on dist-test-slave-r7bh"
server_key: "3af4834f007d210755c1597f0d75c5ec"
server_key_iv: "5f12ce5ae24bb5329c331c895749c9fe"
server_key_version: "encryptionkey@0"
I20260619 00:58:09.355448 21349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 00:58:09.359432 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:09.361073 21349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:58:09.361280 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "d224bfc3134a4e08b644926bb5685607"
format_stamp: "Formatted at 2026-06-19 00:58:09 on dist-test-slave-r7bh"
server_key: "3af4834f007d210755c1597f0d75c5ec"
server_key_iv: "5f12ce5ae24bb5329c331c895749c9fe"
server_key_version: "encryptionkey@0"
I20260619 00:58:09.361501 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:09.367841 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:09.372669 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:09.373099 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:09.392165 21349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45639
I20260619 00:58:09.392166 21415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45639 every 8 connection(s)
I20260619 00:58:09.393643 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:09.397495 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:09.402663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21349
I20260619 00:58:09.402858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:09.403324 388 external_mini_cluster.cc:1428] Setting key 10dea9652a570b2d7feb7355275fefc6
I20260619 00:58:09.410171 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: Bootstrap starting.
I20260619 00:58:09.412791 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:09.405322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45937 (local address 127.0.0.1:45639)
0619 00:58:09.406162 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:09.406184 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:09.406241 (+ 57us) server_negotiation.cc:407] Connection header received
0619 00:58:09.408788 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:09.408849 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:09.409332 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:09.409805 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:09.410817 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:09.410824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:09.410842 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:09.410864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:09.410930 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:09.410955 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:09.411174 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:09.411301 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 00:58:09.411483 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":353,"threads_started":1}
I20260619 00:58:09.414029 21416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:09.415109 21416 log.cc:824] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:09.416771 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:09.418658 21416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: No bootstrap required, opened a new log
I20260619 00:58:09.424633 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:09.421996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47401 (local address 127.0.0.1:45639)
0619 00:58:09.422222 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:09.422232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:09.422640 (+ 408us) server_negotiation.cc:407] Connection header received
0619 00:58:09.422835 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:09.422839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:09.422893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:09.422974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:09.423907 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:09.423911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:09.423914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:09.423928 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:09.423964 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:09.423978 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:09.424096 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:09.424219 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 00:58:09.424266 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:58:09.428543 21416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } }
I20260619 00:58:09.429003 21416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:09.429103 21416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d224bfc3134a4e08b644926bb5685607, State: Initialized, Role: FOLLOWER
I20260619 00:58:09.429733 21416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } }
I20260619 00:58:09.429965 21416 raft_consensus.cc:388] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:09.430123 21416 raft_consensus.cc:482] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:09.430320 21416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:09.432679 21416 raft_consensus.cc:504] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } }
I20260619 00:58:09.433322 21416 leader_election.cc:302] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d224bfc3134a4e08b644926bb5685607; no voters:
I20260619 00:58:09.433946 21416 leader_election.cc:288] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:09.434111 21426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:09.434806 21426 raft_consensus.cc:686] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [term 1 LEADER]: Becoming Leader. State: Replica: d224bfc3134a4e08b644926bb5685607, State: Running, Role: LEADER
I20260619 00:58:09.435326 21426 consensus_queue.cc:237] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } }
I20260619 00:58:09.436121 21416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:09.438274 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d224bfc3134a4e08b644926bb5685607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } } }
I20260619 00:58:09.438691 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:09.439203 21428 sys_catalog.cc:455] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d224bfc3134a4e08b644926bb5685607. Latest consensus state: current_term: 1 leader_uuid: "d224bfc3134a4e08b644926bb5685607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d224bfc3134a4e08b644926bb5685607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45639 } } }
I20260619 00:58:09.439465 21428 sys_catalog.cc:458] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607 [sys.catalog]: This master's current role is: LEADER
W20260619 00:58:09.448486 21442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:09.448625 21442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:09.448771 21431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:09.450999 21431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:09.460373 21431 catalog_manager.cc:1347] Generated new cluster ID: 87ffbd6606c64b988f37bd6bd3a519ab
I20260619 00:58:09.460508 21431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:09.495180 21431 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:09.495671 21431 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:09.505266 21431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d224bfc3134a4e08b644926bb5685607: Generated new TSK 0
I20260619 00:58:09.506366 21431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:09.516566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21349
W20260619 00:58:09.518252 21411 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45937
W20260619 00:58:09.519055 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:09.519224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21349
2026-06-19T00:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:09 dist-test-slave-r7bh krb5kdc[21447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:09 dist-test-slave-r7bh krb5kdc[21447](info): set up 2 sockets
Jun 19 00:58:09 dist-test-slave-r7bh krb5kdc[21447](info): commencing operation
krb5kdc: starting...
W20260619 00:58:10.629374 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:10 dist-test-slave-r7bh krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:10.899734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:11.257157 21460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:11.257450 21460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:11.257547 21460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:11.264922 21460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:11.265060 21460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:11.265115 21460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:11.265180 21460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:11.265262 21460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:11.275666 21460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:11.277122 21460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:11.278417 21460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:11.290603 21465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:11.290659 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:11.290684 21466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:11.291359 21460 server_base.cc:1034] running on GCE node
I20260619 00:58:11.292188 21460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:11.293744 21460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:11.295007 21460 hybrid_clock.cc:648] HybridClock initialized: now 1781830691294923 us; error 68 us; skew 500 ppm
I20260619 00:58:11.295392 21460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:11 dist-test-slave-r7bh krb5kdc[21447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:11.298588 21460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:11.300016 21460 webserver.cc:460] Webserver started at http://127.0.0.1:43555/ using document root <none> and password file <none>
I20260619 00:58:11.300655 21460 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:11.300755 21460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:11.301071 21460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:11.303987 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "0cc4763cf415464bb709a834a93354fa"
format_stamp: "Formatted at 2026-06-19 00:58:11 on dist-test-slave-r7bh"
server_key: "703bccc94abb3b2783cb438297313c37"
server_key_iv: "f16b7a581ed71f7a69e9f027b2cb1d4f"
server_key_version: "encryptionkey@0"
I20260619 00:58:11.304708 21460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "0cc4763cf415464bb709a834a93354fa"
format_stamp: "Formatted at 2026-06-19 00:58:11 on dist-test-slave-r7bh"
server_key: "703bccc94abb3b2783cb438297313c37"
server_key_iv: "f16b7a581ed71f7a69e9f027b2cb1d4f"
server_key_version: "encryptionkey@0"
I20260619 00:58:11.309918 21460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 00:58:11.313282 21475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:11.314671 21460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:58:11.314863 21460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "0cc4763cf415464bb709a834a93354fa"
format_stamp: "Formatted at 2026-06-19 00:58:11 on dist-test-slave-r7bh"
server_key: "703bccc94abb3b2783cb438297313c37"
server_key_iv: "f16b7a581ed71f7a69e9f027b2cb1d4f"
server_key_version: "encryptionkey@0"
I20260619 00:58:11.315068 21460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:11.338009 21460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:11.342729 21460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:11.343053 21460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:11.362015 21460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46405
I20260619 00:58:11.362035 21526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46405 every 8 connection(s)
I20260619 00:58:11.363467 21460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:11.367265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21460
I20260619 00:58:11.367471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:11.367725 21527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:11.367964 388 external_mini_cluster.cc:1428] Setting key 5a11e6e36091110da9e169a8bd1b161d
I20260619 00:58:11.382695 21527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa: Bootstrap starting.
Jun 19 00:58:11 dist-test-slave-r7bh krb5kdc[21447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 00:58:11.386210 21527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:11.387621 21527 log.cc:824] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:11.391252 21527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa: No bootstrap required, opened a new log
I20260619 00:58:11.395432 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:11.370249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:35511 (local address 127.0.0.1:46405)
0619 00:58:11.371313 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:11.371330 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:11.371787 (+ 457us) server_negotiation.cc:407] Connection header received
0619 00:58:11.374517 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:11.374551 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:11.375191 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:11.375709 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:11.377102 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:11.379193 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:11.380553 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:11.381663 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:11.381856 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:11.386304 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:11.386316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:11.386335 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:11.386384 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:11.390509 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:11.391428 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:11.391437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:11.391445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:11.391676 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:11.392141 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:11.392146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:11.392149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:11.392878 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:11.393223 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:11.393650 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 00:58:11.394016 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":457,"threads_started":1}
I20260619 00:58:11.400265 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:11.401799 21527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260619 00:58:11.402259 21527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:11.402351 21527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc4763cf415464bb709a834a93354fa, State: Initialized, Role: FOLLOWER
I20260619 00:58:11.403017 21527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260619 00:58:11.403324 21527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:11.403475 21527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:11.403682 21527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:11.406773 21527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260619 00:58:11.407367 21527 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc4763cf415464bb709a834a93354fa; no voters:
I20260619 00:58:11.408002 21527 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:11.408107 21537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:11.410841 21527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:11.415359 21537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [term 1 LEADER]: Becoming Leader. State: Replica: 0cc4763cf415464bb709a834a93354fa, State: Running, Role: LEADER
I20260619 00:58:11.416186 21537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } }
I20260619 00:58:11.419984 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc4763cf415464bb709a834a93354fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } } }
I20260619 00:58:11.420012 21539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc4763cf415464bb709a834a93354fa. Latest consensus state: current_term: 1 leader_uuid: "0cc4763cf415464bb709a834a93354fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4763cf415464bb709a834a93354fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46405 } } }
I20260619 00:58:11.420504 21539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:11.420506 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:11.421770 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:11.405738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:59411 (local address 127.0.0.1:46405)
0619 00:58:11.405949 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:11.405956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:11.406345 (+ 389us) server_negotiation.cc:407] Connection header received
0619 00:58:11.406489 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:11.406497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:11.406598 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:11.406717 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:11.408212 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:11.409583 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:11.410927 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:11.411407 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:11.411556 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:11.414249 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:11.414254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:11.414257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:11.414289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:11.417431 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:11.418179 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:11.418190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:11.418192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:11.418397 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:11.419026 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:11.419031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:11.419034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:11.419484 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:11.419783 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:11.421200 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0619 00:58:11.421384 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:58:11.425853 21544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:11.433256 21544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:11.442329 21544 catalog_manager.cc:1347] Generated new cluster ID: 1ad14ff64b2f40b0bc741a81257c8dd8
I20260619 00:58:11.442456 21544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:11.457012 21544 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:11.458336 21544 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:11.472512 21544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc4763cf415464bb709a834a93354fa: Generated new TSK 0
I20260619 00:58:11.473448 21544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:11.479600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21460
I20260619 00:58:11.481112 21522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:11.574992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21460
W20260619 00:58:11.593542 21445 connection.cc:535] client connection to 127.0.0.1:46405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46405): Connection reset by peer (error 104)
2026-06-19T00:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:11 dist-test-slave-r7bh krb5kdc[21559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:11 dist-test-slave-r7bh krb5kdc[21559](info): set up 2 sockets
Jun 19 00:58:11 dist-test-slave-r7bh krb5kdc[21559](info): commencing operation
krb5kdc: starting...
W20260619 00:58:12.685467 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:12 dist-test-slave-r7bh krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:12.950271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:13.292086 21572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:13.292389 21572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:13.292553 21572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:13.299943 21572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:13.300071 21572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:13.300163 21572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:13.300279 21572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:13.300340 21572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:13.310578 21572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:13.311913 21572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:13.313375 21572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:13.325277 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:13.325346 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:13.325366 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:13.326602 21572 server_base.cc:1034] running on GCE node
I20260619 00:58:13.327468 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:13.329070 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:13.330319 21572 hybrid_clock.cc:648] HybridClock initialized: now 1781830693330282 us; error 49 us; skew 500 ppm
I20260619 00:58:13.330724 21572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:13 dist-test-slave-r7bh krb5kdc[21559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:13.334034 21572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:13.335772 21572 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
I20260619 00:58:13.336488 21572 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:13.336623 21572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:13.337025 21572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:13.340112 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "1d6bdf162d854f3bace703566304f11f"
format_stamp: "Formatted at 2026-06-19 00:58:13 on dist-test-slave-r7bh"
server_key: "48b31cea6372236e4557841a09b8f5fb"
server_key_iv: "ca5a53eaf9cfc17ce429847143b92bc8"
server_key_version: "encryptionkey@0"
I20260619 00:58:13.340969 21572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d6bdf162d854f3bace703566304f11f"
format_stamp: "Formatted at 2026-06-19 00:58:13 on dist-test-slave-r7bh"
server_key: "48b31cea6372236e4557841a09b8f5fb"
server_key_iv: "ca5a53eaf9cfc17ce429847143b92bc8"
server_key_version: "encryptionkey@0"
I20260619 00:58:13.346360 21572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:58:13.350075 21587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:13.351496 21572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 00:58:13.351706 21572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "1d6bdf162d854f3bace703566304f11f"
format_stamp: "Formatted at 2026-06-19 00:58:13 on dist-test-slave-r7bh"
server_key: "48b31cea6372236e4557841a09b8f5fb"
server_key_iv: "ca5a53eaf9cfc17ce429847143b92bc8"
server_key_version: "encryptionkey@0"
I20260619 00:58:13.351907 21572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:13.378943 21572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:13.383778 21572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:13.384104 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:13.403585 21572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43891
I20260619 00:58:13.403616 21638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43891 every 8 connection(s)
I20260619 00:58:13.405213 21572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:13.406406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21572
I20260619 00:58:13.406592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:13.407114 388 external_mini_cluster.cc:1428] Setting key 629936c0495809446f7dae302392dfd1
I20260619 00:58:13.410516 21639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:13.420699 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:13.409140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:49103 (local address 127.0.0.1:43891)
0619 00:58:13.409942 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:13.409954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:13.409997 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:58:13.411928 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:13.411951 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:13.412348 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:13.412746 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:13.413371 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:13.414685 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:13.416057 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:13.416715 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:13.416935 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:13.418317 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:13.418325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:13.418342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:13.418970 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:13.419046 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:13.419077 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:13.419261 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:13.419555 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 00:58:13.419743 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":344,"threads_started":1}
I20260619 00:58:13.422005 21639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f: Bootstrap starting.
I20260619 00:58:13.424852 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:13.425753 21639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:13.426803 21639 log.cc:824] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:13.430734 21639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f: No bootstrap required, opened a new log
I20260619 00:58:13.436628 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:13.429616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:46991 (local address 127.0.0.1:43891)
0619 00:58:13.429780 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:13.429785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:13.430253 (+ 468us) server_negotiation.cc:407] Connection header received
0619 00:58:13.430483 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:13.430487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:13.430552 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:13.430661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:13.431236 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:13.432120 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:13.433621 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:13.434110 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:13.434286 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:13.435715 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:13.435723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:13.435734 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:13.435762 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:13.435820 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:13.435840 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:13.435973 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:13.436186 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 00:58:13.436253 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:58:13.440026 21639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260619 00:58:13.440435 21639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:13.440555 21639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6bdf162d854f3bace703566304f11f, State: Initialized, Role: FOLLOWER
I20260619 00:58:13.441330 21639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260619 00:58:13.441588 21639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:13.441754 21639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:13.441942 21639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:13.444114 21639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260619 00:58:13.444620 21639 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6bdf162d854f3bace703566304f11f; no voters:
I20260619 00:58:13.445269 21639 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:13.445405 21649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:13.446640 21649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [term 1 LEADER]: Becoming Leader. State: Replica: 1d6bdf162d854f3bace703566304f11f, State: Running, Role: LEADER
I20260619 00:58:13.447307 21649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } }
I20260619 00:58:13.447626 21639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:13.450300 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d6bdf162d854f3bace703566304f11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } } }
I20260619 00:58:13.450419 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6bdf162d854f3bace703566304f11f. Latest consensus state: current_term: 1 leader_uuid: "1d6bdf162d854f3bace703566304f11f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bdf162d854f3bace703566304f11f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43891 } } }
I20260619 00:58:13.450729 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:13.450824 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:13.456660 21657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:13.461426 21657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:13.470332 21657 catalog_manager.cc:1347] Generated new cluster ID: 9914eda61b7b43bfbf0d2117a44abb68
I20260619 00:58:13.470467 21657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:13.493680 21657 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:13.495711 21657 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:13.519899 21657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6bdf162d854f3bace703566304f11f: Generated new TSK 0
I20260619 00:58:13.520774 21657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:13.527532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21572
W20260619 00:58:13.528720 21634 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:49103
W20260619 00:58:13.529553 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:13.529713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21572
W20260619 00:58:13.547338 21557 connection.cc:535] client connection to 127.0.0.1:43891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43891): Connection reset by peer (error 104)
2026-06-19T00:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:13 dist-test-slave-r7bh krb5kdc[21670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:13 dist-test-slave-r7bh krb5kdc[21670](info): set up 2 sockets
Jun 19 00:58:13 dist-test-slave-r7bh krb5kdc[21670](info): commencing operation
krb5kdc: starting...
W20260619 00:58:14.629344 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:14 dist-test-slave-r7bh krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:14.900571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:15.235603 21683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:15.235891 21683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:15.235999 21683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:15.243367 21683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:15.243495 21683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:15.243542 21683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:15.243603 21683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:15.243640 21683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:15.253552 21683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:15.254858 21683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:15.256150 21683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:15.268172 21688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:15.268119 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:15.268136 21691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:15.269153 21683 server_base.cc:1034] running on GCE node
I20260619 00:58:15.270027 21683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:15.271519 21683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:15.272751 21683 hybrid_clock.cc:648] HybridClock initialized: now 1781830695272701 us; error 45 us; skew 500 ppm
I20260619 00:58:15.273135 21683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:15 dist-test-slave-r7bh krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:15.276366 21683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:15.277933 21683 webserver.cc:460] Webserver started at http://127.0.0.1:39291/ using document root <none> and password file <none>
I20260619 00:58:15.278654 21683 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:15.278805 21683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:15.279130 21683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:15.282292 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "2a2903dec9aa4eaa96c91dc958b47717"
format_stamp: "Formatted at 2026-06-19 00:58:15 on dist-test-slave-r7bh"
server_key: "7a2b45ed8c7035a1bab284d31775b5be"
server_key_iv: "b54d1db1220df62b6a660092d1efdb6d"
server_key_version: "encryptionkey@0"
I20260619 00:58:15.283121 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a2903dec9aa4eaa96c91dc958b47717"
format_stamp: "Formatted at 2026-06-19 00:58:15 on dist-test-slave-r7bh"
server_key: "7a2b45ed8c7035a1bab284d31775b5be"
server_key_iv: "b54d1db1220df62b6a660092d1efdb6d"
server_key_version: "encryptionkey@0"
I20260619 00:58:15.288573 21683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 00:58:15.292263 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:15.293735 21683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:58:15.293932 21683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "2a2903dec9aa4eaa96c91dc958b47717"
format_stamp: "Formatted at 2026-06-19 00:58:15 on dist-test-slave-r7bh"
server_key: "7a2b45ed8c7035a1bab284d31775b5be"
server_key_iv: "b54d1db1220df62b6a660092d1efdb6d"
server_key_version: "encryptionkey@0"
I20260619 00:58:15.294147 21683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:15.300153 21683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:15.304100 21683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:15.304394 21683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:15.323536 21683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37169
I20260619 00:58:15.323578 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37169 every 8 connection(s)
I20260619 00:58:15.325028 21683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:15.325799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21683
I20260619 00:58:15.326010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:15.326521 388 external_mini_cluster.cc:1428] Setting key 50016fc7a65a1f8b9098aef93d5f9f94
I20260619 00:58:15.329859 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:15.335593 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:15.328434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:40193 (local address 127.0.0.1:37169)
0619 00:58:15.329261 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:15.329278 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:15.329333 (+ 55us) server_negotiation.cc:407] Connection header received
0619 00:58:15.331726 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:15.331752 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:15.332184 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:15.332629 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:15.333664 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:15.333671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:15.333688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:15.333720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:15.333802 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:15.333828 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:15.334046 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:15.334178 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 00:58:15.334389 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":366,"threads_started":1}
I20260619 00:58:15.339907 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:15.344465 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717: Bootstrap starting.
I20260619 00:58:15.347988 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:15.345296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:54043 (local address 127.0.0.1:37169)
0619 00:58:15.345519 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:15.345528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:15.346044 (+ 516us) server_negotiation.cc:407] Connection header received
0619 00:58:15.346218 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:15.346222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:15.346275 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:15.346363 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:15.347319 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:15.347323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:15.347326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:15.347340 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:15.347380 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:15.347395 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:15.347486 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:15.347617 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 00:58:15.347668 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:58:15.348737 21750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:15.350086 21750 log.cc:824] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:15.353134 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717: No bootstrap required, opened a new log
I20260619 00:58:15.361109 21750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260619 00:58:15.361573 21750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:15.361694 21750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2903dec9aa4eaa96c91dc958b47717, State: Initialized, Role: FOLLOWER
I20260619 00:58:15.362358 21750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260619 00:58:15.362687 21750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:15.362855 21750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:15.363066 21750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:15.365427 21750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260619 00:58:15.365950 21750 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2903dec9aa4eaa96c91dc958b47717; no voters:
I20260619 00:58:15.366521 21750 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:15.366684 21760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:15.367396 21760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2903dec9aa4eaa96c91dc958b47717, State: Running, Role: LEADER
I20260619 00:58:15.367910 21760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260619 00:58:15.368670 21750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:15.370898 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2903dec9aa4eaa96c91dc958b47717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260619 00:58:15.370927 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2903dec9aa4eaa96c91dc958b47717. Latest consensus state: current_term: 1 leader_uuid: "2a2903dec9aa4eaa96c91dc958b47717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2903dec9aa4eaa96c91dc958b47717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260619 00:58:15.371408 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:15.371408 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:15.373288 21765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:15.378618 21765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:15.387428 21765 catalog_manager.cc:1347] Generated new cluster ID: 9a06edd92a214a3ca74093638d1c6db2
I20260619 00:58:15.387544 21765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:15.420217 21765 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:15.420734 21765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:15.435078 21765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2903dec9aa4eaa96c91dc958b47717: Generated new TSK 0
I20260619 00:58:15.436024 21765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:15.443634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21683
W20260619 00:58:15.444994 21745 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:40193
W20260619 00:58:15.445825 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:15.446089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21683
2026-06-19T00:58:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:15 dist-test-slave-r7bh krb5kdc[21781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:15 dist-test-slave-r7bh krb5kdc[21781](info): set up 2 sockets
Jun 19 00:58:15 dist-test-slave-r7bh krb5kdc[21781](info): commencing operation
krb5kdc: starting...
W20260619 00:58:16.577565 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:16 dist-test-slave-r7bh krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:16.843920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:17.183725 21794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:17.184041 21794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:17.184263 21794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:17.191576 21794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:17.191694 21794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:17.191788 21794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:17.191911 21794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:17.191972 21794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:17.202013 21794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:17.203405 21794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:17.204779 21794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:17.217064 21799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:17.217089 21800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:17.217089 21802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:17.218199 21794 server_base.cc:1034] running on GCE node
I20260619 00:58:17.219079 21794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:17.220582 21794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:17.221860 21794 hybrid_clock.cc:648] HybridClock initialized: now 1781830697221766 us; error 105 us; skew 500 ppm
I20260619 00:58:17.222283 21794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:17 dist-test-slave-r7bh krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:17.230070 21794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:17.231940 21794 webserver.cc:460] Webserver started at http://127.0.0.1:43047/ using document root <none> and password file <none>
I20260619 00:58:17.232720 21794 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:17.232894 21794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:17.233276 21794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:17.236476 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "df928882012c4859b588972f38170ba5"
format_stamp: "Formatted at 2026-06-19 00:58:17 on dist-test-slave-r7bh"
server_key: "ce069765b70693f43d7c650f531c0726"
server_key_iv: "27a4dc57163d6f513bd12a67a28bfb26"
server_key_version: "encryptionkey@0"
I20260619 00:58:17.237314 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "df928882012c4859b588972f38170ba5"
format_stamp: "Formatted at 2026-06-19 00:58:17 on dist-test-slave-r7bh"
server_key: "ce069765b70693f43d7c650f531c0726"
server_key_iv: "27a4dc57163d6f513bd12a67a28bfb26"
server_key_version: "encryptionkey@0"
I20260619 00:58:17.242864 21794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 00:58:17.246743 21809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:17.248176 21794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 00:58:17.248380 21794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "df928882012c4859b588972f38170ba5"
format_stamp: "Formatted at 2026-06-19 00:58:17 on dist-test-slave-r7bh"
server_key: "ce069765b70693f43d7c650f531c0726"
server_key_iv: "27a4dc57163d6f513bd12a67a28bfb26"
server_key_version: "encryptionkey@0"
I20260619 00:58:17.248602 21794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:17.284658 21794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:17.289204 21794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:17.289531 21794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:17.308998 21794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38967
I20260619 00:58:17.309015 21860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38967 every 8 connection(s)
I20260619 00:58:17.310541 21794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:17.314725 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:17.320385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21794
I20260619 00:58:17.320590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:17.321106 388 external_mini_cluster.cc:1428] Setting key e42cbd4f9d2cb9de17564f2579362d0c
I20260619 00:58:17.327167 21861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5: Bootstrap starting.
I20260619 00:58:17.330536 21861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:17.331557 21861 log.cc:824] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:17.334409 21861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5: No bootstrap required, opened a new log
Jun 19 00:58:17 dist-test-slave-r7bh krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 00:58:17.345299 21861 raft_consensus.cc:348] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260619 00:58:17.345841 21861 raft_consensus.cc:374] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:17.345875 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:17.323244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45427 (local address 127.0.0.1:38967)
0619 00:58:17.324082 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:17.324095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:17.324142 (+ 47us) server_negotiation.cc:407] Connection header received
0619 00:58:17.326067 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:17.326104 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:17.326569 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:17.326976 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:17.328456 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:17.329873 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:17.331234 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:17.331970 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:17.332213 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:17.336366 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:17.336377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:17.336397 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:17.336457 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:17.340661 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:17.341601 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:17.341614 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:17.341624 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:17.341845 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:17.342325 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:17.342330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:17.342334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:17.343170 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:17.343529 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:17.343987 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 00:58:17.344368 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":341,"threads_started":1}
I20260619 00:58:17.345980 21861 raft_consensus.cc:729] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df928882012c4859b588972f38170ba5, State: Initialized, Role: FOLLOWER
I20260619 00:58:17.346599 21861 consensus_queue.cc:260] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260619 00:58:17.346879 21861 raft_consensus.cc:388] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:17.347033 21861 raft_consensus.cc:482] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:17.347215 21861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:17.350458 21861 raft_consensus.cc:504] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260619 00:58:17.350745 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:17.351097 21861 leader_election.cc:302] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df928882012c4859b588972f38170ba5; no voters:
I20260619 00:58:17.351766 21861 leader_election.cc:288] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:17.351888 21866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:17.354007 21861 sys_catalog.cc:564] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:17.356768 21866 raft_consensus.cc:686] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [term 1 LEADER]: Becoming Leader. State: Replica: df928882012c4859b588972f38170ba5, State: Running, Role: LEADER
I20260619 00:58:17.357529 21866 consensus_queue.cc:237] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260619 00:58:17.360368 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df928882012c4859b588972f38170ba5. Latest consensus state: current_term: 1 leader_uuid: "df928882012c4859b588972f38170ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260619 00:58:17.360450 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df928882012c4859b588972f38170ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df928882012c4859b588972f38170ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260619 00:58:17.360827 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:17.360918 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:17.365515 21876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:17.371976 21876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:17.379431 21888 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 00:58:17.384917 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:17.357392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:33517 (local address 127.0.0.1:38967)
0619 00:58:17.357588 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:17.357594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:17.366031 (+ 8437us) server_negotiation.cc:407] Connection header received
0619 00:58:17.366328 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:17.366334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:17.366423 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:17.366517 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:17.368804 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:17.370022 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:17.371863 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:17.372575 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:17.372708 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:17.377072 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:17.377083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:17.377090 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:17.377147 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:17.381511 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:17.382321 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:17.382328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:17.382331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:17.382467 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:17.383040 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:17.383045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:17.383049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:17.383406 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:17.383679 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:17.384187 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 00:58:17.384379 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:58:17.391409 21876 catalog_manager.cc:1347] Generated new cluster ID: eb53b39b1c59469081ad3ba81824ffac
I20260619 00:58:17.391530 21876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:17.413077 21876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:17.414672 21876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:17.420567 21876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df928882012c4859b588972f38170ba5: Generated new TSK 0
I20260619 00:58:17.421439 21876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:17.428287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21794
I20260619 00:58:17.429792 21856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:17.524860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21794
W20260619 00:58:17.542878 21779 connection.cc:535] client connection to 127.0.0.1:38967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38967): Connection reset by peer (error 104)
2026-06-19T00:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:17 dist-test-slave-r7bh krb5kdc[21893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:17 dist-test-slave-r7bh krb5kdc[21893](info): set up 2 sockets
Jun 19 00:58:17 dist-test-slave-r7bh krb5kdc[21893](info): commencing operation
krb5kdc: starting...
W20260619 00:58:18.641386 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:18 dist-test-slave-r7bh krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:18.901723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:19.235666 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:19.235972 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:19.236068 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:19.243201 21906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:19.243325 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:19.243372 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:19.243434 21906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:19.243479 21906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:19.253450 21906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:19.254701 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:19.255961 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:19.268035 21911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:19.267926 21914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:19.268050 21912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:19.269078 21906 server_base.cc:1034] running on GCE node
I20260619 00:58:19.269834 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:19.271351 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:19.272606 21906 hybrid_clock.cc:648] HybridClock initialized: now 1781830699272536 us; error 56 us; skew 500 ppm
I20260619 00:58:19.273070 21906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:19 dist-test-slave-r7bh krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:19.276559 21906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:19.278144 21906 webserver.cc:460] Webserver started at http://127.0.0.1:37491/ using document root <none> and password file <none>
I20260619 00:58:19.278890 21906 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:19.279032 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:19.279358 21906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:19.282585 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "ad775a5da32947a59dbb7d5308c32c9b"
format_stamp: "Formatted at 2026-06-19 00:58:19 on dist-test-slave-r7bh"
server_key: "6b15a6d4b77cb76b0ccf3b4ad0b7676c"
server_key_iv: "daf70f76d784e25721dcd423ed18c5f9"
server_key_version: "encryptionkey@0"
I20260619 00:58:19.283420 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad775a5da32947a59dbb7d5308c32c9b"
format_stamp: "Formatted at 2026-06-19 00:58:19 on dist-test-slave-r7bh"
server_key: "6b15a6d4b77cb76b0ccf3b4ad0b7676c"
server_key_iv: "daf70f76d784e25721dcd423ed18c5f9"
server_key_version: "encryptionkey@0"
I20260619 00:58:19.289362 21906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 00:58:19.293066 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:19.294625 21906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:19.294848 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "ad775a5da32947a59dbb7d5308c32c9b"
format_stamp: "Formatted at 2026-06-19 00:58:19 on dist-test-slave-r7bh"
server_key: "6b15a6d4b77cb76b0ccf3b4ad0b7676c"
server_key_iv: "daf70f76d784e25721dcd423ed18c5f9"
server_key_version: "encryptionkey@0"
I20260619 00:58:19.295074 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:19.311957 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:19.316402 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:19.316720 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:19.336643 21906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45513
I20260619 00:58:19.336695 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45513 every 8 connection(s)
I20260619 00:58:19.338225 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:19.342268 21973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:19.348085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 21906
I20260619 00:58:19.348299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:19.348698 388 external_mini_cluster.cc:1428] Setting key 413f8cfe9d569d4126e51160fa9d4d46
I20260619 00:58:19.353345 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b: Bootstrap starting.
I20260619 00:58:19.356977 21973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:19.358224 21973 log.cc:824] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:19.361701 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b: No bootstrap required, opened a new log
I20260619 00:58:19.362965 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:19.350680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:33881 (local address 127.0.0.1:45513)
0619 00:58:19.351527 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:19.351539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:19.351581 (+ 42us) server_negotiation.cc:407] Connection header received
0619 00:58:19.353377 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:19.353400 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:19.353854 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:19.354176 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:19.354795 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:19.356124 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:19.358078 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:19.358993 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:19.359196 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:19.360354 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:19.360361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:19.360376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:19.360992 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:19.361076 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:19.361103 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:19.361291 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:19.361651 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 00:58:19.361855 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":398,"threads_started":1}
I20260619 00:58:19.367486 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:19.370908 21973 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260619 00:58:19.371460 21973 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:19.371611 21973 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad775a5da32947a59dbb7d5308c32c9b, State: Initialized, Role: FOLLOWER
I20260619 00:58:19.372330 21973 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260619 00:58:19.372617 21973 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:19.372805 21973 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:19.373106 21973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:19.376214 21973 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260619 00:58:19.376776 21973 leader_election.cc:302] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad775a5da32947a59dbb7d5308c32c9b; no voters:
I20260619 00:58:19.377548 21973 leader_election.cc:288] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:19.377769 21983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:19.378629 21983 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [term 1 LEADER]: Becoming Leader. State: Replica: ad775a5da32947a59dbb7d5308c32c9b, State: Running, Role: LEADER
I20260619 00:58:19.379388 21983 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260619 00:58:19.380546 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:19.373104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:58257 (local address 127.0.0.1:45513)
0619 00:58:19.373311 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:19.373317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:19.373908 (+ 591us) server_negotiation.cc:407] Connection header received
0619 00:58:19.374059 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:19.374064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:19.374151 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:19.374266 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:19.374700 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:19.375865 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:19.377325 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:19.377984 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:19.378131 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:19.379565 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:19.379571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:19.379575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:19.379597 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:19.379691 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:19.379716 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:19.379793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:19.380136 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 00:58:19.380192 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:58:19.381138 21973 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:19.382171 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad775a5da32947a59dbb7d5308c32c9b. Latest consensus state: current_term: 1 leader_uuid: "ad775a5da32947a59dbb7d5308c32c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } } }
I20260619 00:58:19.382555 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:19.383205 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad775a5da32947a59dbb7d5308c32c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad775a5da32947a59dbb7d5308c32c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } } }
I20260619 00:58:19.383533 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:19.388854 21988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:19.393759 21988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:19.402743 21988 catalog_manager.cc:1347] Generated new cluster ID: 54c784980b3e4af3a1adccc565b944d3
I20260619 00:58:19.402895 21988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:19.452124 21988 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:19.453613 21988 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:19.463131 21988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad775a5da32947a59dbb7d5308c32c9b: Generated new TSK 0
I20260619 00:58:19.464308 21988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:19.480245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21906
W20260619 00:58:19.482344 21968 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:33881
W20260619 00:58:19.483219 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 21906: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:19.483405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 21906
W20260619 00:58:19.502168 21891 connection.cc:535] client connection to 127.0.0.1:45513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45513): Connection reset by peer (error 104)
2026-06-19T00:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:19 dist-test-slave-r7bh krb5kdc[22005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:19 dist-test-slave-r7bh krb5kdc[22005](info): set up 2 sockets
Jun 19 00:58:19 dist-test-slave-r7bh krb5kdc[22005](info): commencing operation
krb5kdc: starting...
W20260619 00:58:20.601215 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:20 dist-test-slave-r7bh krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 00:58:20.857826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:21.206067 22018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:21.206398 22018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:21.206720 22018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:21.214833 22018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:21.214946 22018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:21.214994 22018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:21.215078 22018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:21.215168 22018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:21.225472 22018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:21.226781 22018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:21.228117 22018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:21.240168 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:21.240332 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:21.240873 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:21.242457 22018 server_base.cc:1034] running on GCE node
I20260619 00:58:21.243630 22018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:21.245644 22018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:21.246879 22018 hybrid_clock.cc:648] HybridClock initialized: now 1781830701246817 us; error 48 us; skew 500 ppm
I20260619 00:58:21.247267 22018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:21 dist-test-slave-r7bh krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:21.252065 22018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 00:58:21.256116 22018 webserver.cc:460] Webserver started at http://127.0.0.1:37639/ using document root <none> and password file <none>
I20260619 00:58:21.257103 22018 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:21.257249 22018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:21.257553 22018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:21.262204 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "2223b5120f5348bda6ab69230b3cc248"
format_stamp: "Formatted at 2026-06-19 00:58:21 on dist-test-slave-r7bh"
server_key: "869680298903936285e7de8702609ee2"
server_key_iv: "f69e5a8bce7a6a03ce1fb07211930f8a"
server_key_version: "encryptionkey@0"
I20260619 00:58:21.263219 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "2223b5120f5348bda6ab69230b3cc248"
format_stamp: "Formatted at 2026-06-19 00:58:21 on dist-test-slave-r7bh"
server_key: "869680298903936285e7de8702609ee2"
server_key_iv: "f69e5a8bce7a6a03ce1fb07211930f8a"
server_key_version: "encryptionkey@0"
I20260619 00:58:21.271169 22018 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260619 00:58:21.275673 22033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:21.277387 22018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:58:21.277573 22018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "2223b5120f5348bda6ab69230b3cc248"
format_stamp: "Formatted at 2026-06-19 00:58:21 on dist-test-slave-r7bh"
server_key: "869680298903936285e7de8702609ee2"
server_key_iv: "f69e5a8bce7a6a03ce1fb07211930f8a"
server_key_version: "encryptionkey@0"
I20260619 00:58:21.277807 22018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:21.286355 22018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:21.291617 22018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:21.291940 22018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:21.310448 22018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42771
I20260619 00:58:21.310495 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42771 every 8 connection(s)
I20260619 00:58:21.311842 22018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:21.312597 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22018
I20260619 00:58:21.312834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:21.313364 388 external_mini_cluster.cc:1428] Setting key acbcaa03a329b948afcdf4ad284ab4c8
I20260619 00:58:21.316385 22085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:21.322435 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:21.315279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:49949 (local address 127.0.0.1:42771)
0619 00:58:21.316062 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:21.316077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:21.316126 (+ 49us) server_negotiation.cc:407] Connection header received
0619 00:58:21.318533 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:21.318560 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:21.318996 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:21.319437 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:21.320456 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:21.320464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:21.320480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:21.320498 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:21.320580 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:21.320611 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:21.320887 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:21.321015 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 00:58:21.321218 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":302,"threads_started":1}
I20260619 00:58:21.326112 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:21.329128 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248: Bootstrap starting.
I20260619 00:58:21.332752 22085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:21.333082 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:21.330454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:48911 (local address 127.0.0.1:42771)
0619 00:58:21.330669 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:21.330677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:21.330973 (+ 296us) server_negotiation.cc:407] Connection header received
0619 00:58:21.331106 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:21.331112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:21.331175 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:21.331299 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:21.332355 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:21.332360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:21.332365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:21.332382 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:21.332430 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:21.332450 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:21.332574 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:21.332689 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 00:58:21.332760 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:58:21.334139 22085 log.cc:824] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:21.337005 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248: No bootstrap required, opened a new log
I20260619 00:58:21.344622 22085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } }
I20260619 00:58:21.345077 22085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:21.345165 22085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2223b5120f5348bda6ab69230b3cc248, State: Initialized, Role: FOLLOWER
I20260619 00:58:21.345772 22085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } }
I20260619 00:58:21.345975 22085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:21.346076 22085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:21.346206 22085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:21.348956 22085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } }
I20260619 00:58:21.349444 22085 leader_election.cc:302] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2223b5120f5348bda6ab69230b3cc248; no voters:
I20260619 00:58:21.350059 22085 leader_election.cc:288] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:21.350240 22095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:21.350937 22095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [term 1 LEADER]: Becoming Leader. State: Replica: 2223b5120f5348bda6ab69230b3cc248, State: Running, Role: LEADER
I20260619 00:58:21.351459 22095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } }
I20260619 00:58:21.352228 22085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:21.355221 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2223b5120f5348bda6ab69230b3cc248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } } }
I20260619 00:58:21.355465 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2223b5120f5348bda6ab69230b3cc248. Latest consensus state: current_term: 1 leader_uuid: "2223b5120f5348bda6ab69230b3cc248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2223b5120f5348bda6ab69230b3cc248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42771 } } }
I20260619 00:58:21.355790 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:21.355790 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:21.356545 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:21.361950 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:21.370906 22103 catalog_manager.cc:1347] Generated new cluster ID: 5a5718fb82f346abb845e8984bd8df1d
I20260619 00:58:21.371029 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:21.383539 22103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:21.384181 22103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:21.394975 22103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2223b5120f5348bda6ab69230b3cc248: Generated new TSK 0
I20260619 00:58:21.395929 22103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:21.407253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22018
W20260619 00:58:21.409313 22080 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:49949
W20260619 00:58:21.410252 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22018: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:21.410429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22018
2026-06-19T00:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:21 dist-test-slave-r7bh krb5kdc[22116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:21 dist-test-slave-r7bh krb5kdc[22116](info): set up 2 sockets
Jun 19 00:58:21 dist-test-slave-r7bh krb5kdc[22116](info): commencing operation
krb5kdc: starting...
W20260619 00:58:22.529359 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:22 dist-test-slave-r7bh krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:22.791663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38505
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:23.129113 22129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:23.129393 22129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:23.129487 22129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:23.137333 22129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:23.137444 22129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:23.137494 22129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:23.137558 22129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:23.137600 22129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:23.147393 22129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:23.148762 22129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:23.150192 22129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:23.161844 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:23.161968 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:23.162091 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:23.162910 22129 server_base.cc:1034] running on GCE node
I20260619 00:58:23.163688 22129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:23.165141 22129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:23.166338 22129 hybrid_clock.cc:648] HybridClock initialized: now 1781830703166285 us; error 51 us; skew 500 ppm
I20260619 00:58:23.166714 22129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:23 dist-test-slave-r7bh krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:23.173468 22129 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:23.174978 22129 webserver.cc:460] Webserver started at http://127.0.97.62:44165/ using document root <none> and password file <none>
I20260619 00:58:23.175659 22129 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:23.175800 22129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:23.176098 22129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:23.179116 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "4589fb307dd84995bff2a109cc184b84"
format_stamp: "Formatted at 2026-06-19 00:58:23 on dist-test-slave-r7bh"
server_key: "aae208918cf9684d38ede1fed36ae7f1"
server_key_iv: "12c5fb838fab0bb2ccf9afadebc29cad"
server_key_version: "encryptionkey@0"
I20260619 00:58:23.179900 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "4589fb307dd84995bff2a109cc184b84"
format_stamp: "Formatted at 2026-06-19 00:58:23 on dist-test-slave-r7bh"
server_key: "aae208918cf9684d38ede1fed36ae7f1"
server_key_iv: "12c5fb838fab0bb2ccf9afadebc29cad"
server_key_version: "encryptionkey@0"
I20260619 00:58:23.185379 22129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 00:58:23.189054 22144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:23.190454 22129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:23.190690 22129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "4589fb307dd84995bff2a109cc184b84"
format_stamp: "Formatted at 2026-06-19 00:58:23 on dist-test-slave-r7bh"
server_key: "aae208918cf9684d38ede1fed36ae7f1"
server_key_iv: "12c5fb838fab0bb2ccf9afadebc29cad"
server_key_version: "encryptionkey@0"
I20260619 00:58:23.190908 22129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:23.211814 22129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:23.216346 22129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:23.216647 22129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:23.235263 22129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38505
I20260619 00:58:23.235298 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38505 every 8 connection(s)
I20260619 00:58:23.236651 22129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:23.240346 22196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:23.246958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22129
I20260619 00:58:23.247154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:23.247592 388 external_mini_cluster.cc:1428] Setting key 80c822bba6d3426712c7cbd4f940cddb
I20260619 00:58:23.251139 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84: Bootstrap starting.
I20260619 00:58:23.254205 22196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:23.255242 22196 log.cc:824] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:23.258369 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84: No bootstrap required, opened a new log
Jun 19 00:58:23 dist-test-slave-r7bh krb5kdc[22116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:58:23.267102 22196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } }
I20260619 00:58:23.267539 22196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:23.267650 22196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4589fb307dd84995bff2a109cc184b84, State: Initialized, Role: FOLLOWER
I20260619 00:58:23.268438 22196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } }
I20260619 00:58:23.268790 22196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:23.268983 22196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:23.269176 22196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:23.271749 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:23.249459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60147 (local address 127.0.97.62:38505)
0619 00:58:23.250287 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:23.250300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:23.250345 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:58:23.252259 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:23.252283 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:23.252724 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:23.253187 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:23.254752 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:23.256227 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:23.257747 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:23.258537 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:23.258763 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:23.263419 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:23.263428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:23.263448 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:23.263494 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:23.266867 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:23.267823 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:23.267835 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:23.267843 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:23.268069 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:23.268656 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:23.268660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:23.268662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:23.269450 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:23.269722 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:23.270266 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 00:58:23.270686 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":378,"threads_started":1}
I20260619 00:58:23.272011 22196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } }
I20260619 00:58:23.272589 22196 leader_election.cc:302] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4589fb307dd84995bff2a109cc184b84; no voters:
I20260619 00:58:23.273330 22196 leader_election.cc:288] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:23.273450 22201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:23.274534 22201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [term 1 LEADER]: Becoming Leader. State: Replica: 4589fb307dd84995bff2a109cc184b84, State: Running, Role: LEADER
I20260619 00:58:23.275135 22201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } }
I20260619 00:58:23.276124 22196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:23.276330 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:23.278015 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4589fb307dd84995bff2a109cc184b84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } } }
I20260619 00:58:23.278486 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:23.282212 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4589fb307dd84995bff2a109cc184b84. Latest consensus state: current_term: 1 leader_uuid: "4589fb307dd84995bff2a109cc184b84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4589fb307dd84995bff2a109cc184b84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38505 } } }
I20260619 00:58:23.282598 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:23.293270 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:23.302211 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:23.320407 22206 catalog_manager.cc:1347] Generated new cluster ID: 0ac4e9be4722484ca3c7ac70c3aa6d95
I20260619 00:58:23.320549 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:23.321779 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:23.301023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46631 (local address 127.0.97.62:38505)
0619 00:58:23.301208 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:23.301214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:23.301854 (+ 640us) server_negotiation.cc:407] Connection header received
0619 00:58:23.302196 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:23.302206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:23.302321 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:23.302423 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:23.304934 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:23.306096 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:23.308021 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:23.308702 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:23.308839 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:23.312921 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:23.312931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:23.312939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:23.312996 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:23.318135 (+ 5139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:23.319144 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:23.319150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:23.319153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:23.319331 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:23.319926 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:23.319931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:23.319934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:23.320371 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:23.320636 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:23.321217 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 00:58:23.321399 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:58:23.330698 22206 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:23.332051 22206 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:23.342854 22206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4589fb307dd84995bff2a109cc184b84: Generated new TSK 0
I20260619 00:58:23.343814 22206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:23.356848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22129
I20260619 00:58:23.358011 22191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:23.448884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22129
W20260619 00:58:23.466297 22114 connection.cc:535] client connection to 127.0.97.62:38505 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38505): Connection reset by peer (error 104)
2026-06-19T00:58:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:23 dist-test-slave-r7bh krb5kdc[22229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:23 dist-test-slave-r7bh krb5kdc[22229](info): set up 2 sockets
Jun 19 00:58:23 dist-test-slave-r7bh krb5kdc[22229](info): commencing operation
krb5kdc: starting...
W20260619 00:58:24.553401 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:24 dist-test-slave-r7bh krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:24.820060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32859
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:25.164012 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:25.164386 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:25.164486 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:25.171710 22242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:25.171898 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:25.172004 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:25.172111 22242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:25.172187 22242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:25.182315 22242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:25.183667 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:25.185192 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:25.197376 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:25.197465 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:25.197472 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:25.198887 22242 server_base.cc:1034] running on GCE node
I20260619 00:58:25.199699 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:25.201167 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:25.202399 22242 hybrid_clock.cc:648] HybridClock initialized: now 1781830705202336 us; error 54 us; skew 500 ppm
I20260619 00:58:25.202746 22242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:25 dist-test-slave-r7bh krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:25.205858 22242 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:25.207305 22242 webserver.cc:460] Webserver started at http://127.0.97.62:36797/ using document root <none> and password file <none>
I20260619 00:58:25.207983 22242 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:25.208153 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:25.208493 22242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:25.211506 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "d22f1af636d6456897a03491463723a4"
format_stamp: "Formatted at 2026-06-19 00:58:25 on dist-test-slave-r7bh"
server_key: "38828a1c60d3e27ad10f69ed8fd09ca0"
server_key_iv: "d0b61003f357de9b207ba43ac6dfacee"
server_key_version: "encryptionkey@0"
I20260619 00:58:25.212282 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "d22f1af636d6456897a03491463723a4"
format_stamp: "Formatted at 2026-06-19 00:58:25 on dist-test-slave-r7bh"
server_key: "38828a1c60d3e27ad10f69ed8fd09ca0"
server_key_iv: "d0b61003f357de9b207ba43ac6dfacee"
server_key_version: "encryptionkey@0"
I20260619 00:58:25.217523 22242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:58:25.221014 22257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:25.222307 22242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:25.222487 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "d22f1af636d6456897a03491463723a4"
format_stamp: "Formatted at 2026-06-19 00:58:25 on dist-test-slave-r7bh"
server_key: "38828a1c60d3e27ad10f69ed8fd09ca0"
server_key_iv: "d0b61003f357de9b207ba43ac6dfacee"
server_key_version: "encryptionkey@0"
I20260619 00:58:25.222684 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:25.253818 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:25.257906 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:25.258143 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:25.276496 22242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32859
I20260619 00:58:25.276520 22308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32859 every 8 connection(s)
I20260619 00:58:25.277949 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:25.281934 22309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:25.286536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22242
I20260619 00:58:25.286721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:25.287264 388 external_mini_cluster.cc:1428] Setting key 12a8a0364af9c850fb2543c7a5fab68a
I20260619 00:58:25.294083 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4: Bootstrap starting.
I20260619 00:58:25.298005 22309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:25.299386 22309 log.cc:824] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:25.302927 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4: No bootstrap required, opened a new log
I20260619 00:58:25.303053 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:25.289226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46663 (local address 127.0.97.62:32859)
0619 00:58:25.290071 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:25.290083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:25.290125 (+ 42us) server_negotiation.cc:407] Connection header received
0619 00:58:25.292488 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:25.292514 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:25.293183 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:25.293544 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:25.294246 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:25.295639 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:25.297622 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:25.298442 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:25.298585 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:25.300127 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:25.300148 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:25.300174 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:25.301040 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:25.301161 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:25.301187 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:25.301315 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:25.301723 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 00:58:25.301939 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":391,"threads_started":1}
I20260619 00:58:25.307421 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:25.313113 22309 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } }
I20260619 00:58:25.313614 22309 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:25.313764 22309 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22f1af636d6456897a03491463723a4, State: Initialized, Role: FOLLOWER
I20260619 00:58:25.314531 22309 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } }
I20260619 00:58:25.314852 22309 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:25.315055 22309 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:25.315263 22309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:25.318480 22309 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } }
I20260619 00:58:25.319139 22309 leader_election.cc:302] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22f1af636d6456897a03491463723a4; no voters:
I20260619 00:58:25.319936 22309 leader_election.cc:288] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:25.320042 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:25.320473 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:25.312968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53911 (local address 127.0.97.62:32859)
0619 00:58:25.313190 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:25.313197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:25.313667 (+ 470us) server_negotiation.cc:407] Connection header received
0619 00:58:25.313913 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:25.313919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:25.314000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:25.314125 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:25.314731 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:25.315701 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:25.317620 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:25.318077 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:25.318174 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:25.319482 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:25.319487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:25.319490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:25.319515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:25.319607 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:25.319624 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:25.319772 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:25.319982 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 00:58:25.320052 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:58:25.321753 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [term 1 LEADER]: Becoming Leader. State: Replica: d22f1af636d6456897a03491463723a4, State: Running, Role: LEADER
I20260619 00:58:25.322355 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } }
I20260619 00:58:25.322851 22309 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:25.325850 22320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22f1af636d6456897a03491463723a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } } }
I20260619 00:58:25.326413 22320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:25.325846 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22f1af636d6456897a03491463723a4. Latest consensus state: current_term: 1 leader_uuid: "d22f1af636d6456897a03491463723a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22f1af636d6456897a03491463723a4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32859 } } }
I20260619 00:58:25.326918 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:25.329309 22326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:25.334298 22326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:25.343288 22326 catalog_manager.cc:1347] Generated new cluster ID: 36c698a3ba1b4e7894b4952c3b033c9d
I20260619 00:58:25.343425 22326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:25.350417 22326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:25.352329 22326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:25.363093 22326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22f1af636d6456897a03491463723a4: Generated new TSK 0
I20260619 00:58:25.364303 22326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:25.378762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22242
W20260619 00:58:25.380007 22304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46663
W20260619 00:58:25.380875 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:25.381047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22242
W20260619 00:58:25.398687 22227 connection.cc:535] client connection to 127.0.97.62:32859 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:32859): Connection reset by peer (error 104)
2026-06-19T00:58:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:25 dist-test-slave-r7bh krb5kdc[22340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:25 dist-test-slave-r7bh krb5kdc[22340](info): set up 2 sockets
Jun 19 00:58:25 dist-test-slave-r7bh krb5kdc[22340](info): commencing operation
krb5kdc: starting...
W20260619 00:58:26.485343 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:26 dist-test-slave-r7bh krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:26.747725 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41387
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:27.092849 22353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:27.093218 22353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:27.093384 22353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:27.101408 22353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:27.101562 22353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:27.101667 22353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:27.101775 22353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:27.101850 22353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:27.112011 22353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:27.113428 22353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:27.114777 22353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:27.126780 22361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:27.126830 22359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:27.126865 22358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:27.127703 22353 server_base.cc:1034] running on GCE node
I20260619 00:58:27.128548 22353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:27.130070 22353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:27.131305 22353 hybrid_clock.cc:648] HybridClock initialized: now 1781830707131253 us; error 50 us; skew 500 ppm
I20260619 00:58:27.131672 22353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:27 dist-test-slave-r7bh krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:27.134855 22353 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:27.136356 22353 webserver.cc:460] Webserver started at http://127.0.97.62:35763/ using document root <none> and password file <none>
I20260619 00:58:27.137122 22353 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:27.137223 22353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:27.137485 22353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:27.140411 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "c1d863b6ebfa499887e6af810de2afc1"
format_stamp: "Formatted at 2026-06-19 00:58:27 on dist-test-slave-r7bh"
server_key: "255aefa6bcade5dd4ccd87a5a62ae198"
server_key_iv: "a05fa0032a9eb4ee016187fd311a75ba"
server_key_version: "encryptionkey@0"
I20260619 00:58:27.141222 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "c1d863b6ebfa499887e6af810de2afc1"
format_stamp: "Formatted at 2026-06-19 00:58:27 on dist-test-slave-r7bh"
server_key: "255aefa6bcade5dd4ccd87a5a62ae198"
server_key_iv: "a05fa0032a9eb4ee016187fd311a75ba"
server_key_version: "encryptionkey@0"
I20260619 00:58:27.146790 22353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:58:27.150367 22368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:27.151682 22353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:27.151872 22353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "c1d863b6ebfa499887e6af810de2afc1"
format_stamp: "Formatted at 2026-06-19 00:58:27 on dist-test-slave-r7bh"
server_key: "255aefa6bcade5dd4ccd87a5a62ae198"
server_key_iv: "a05fa0032a9eb4ee016187fd311a75ba"
server_key_version: "encryptionkey@0"
I20260619 00:58:27.152076 22353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:27.157820 22353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:27.162515 22353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:27.162819 22353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:27.181387 22353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41387
I20260619 00:58:27.181471 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41387 every 8 connection(s)
I20260619 00:58:27.182807 22353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:27.183341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22353
I20260619 00:58:27.183562 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:27.184142 388 external_mini_cluster.cc:1428] Setting key 0f70c58c9687cff766e7ad8f8c00cbb2
I20260619 00:58:27.187302 22420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:27.191794 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:27.185921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37157 (local address 127.0.97.62:41387)
0619 00:58:27.186685 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:27.186696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:27.186736 (+ 40us) server_negotiation.cc:407] Connection header received
0619 00:58:27.188372 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:27.188393 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:27.188748 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:27.189182 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:27.190229 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:27.190235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:27.190248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:27.190261 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:27.190322 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:27.190341 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:27.190572 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:27.190714 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 00:58:27.190886 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":336,"threads_started":1}
I20260619 00:58:27.195447 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:27.198649 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1: Bootstrap starting.
I20260619 00:58:27.201790 22420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:27.202128 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:27.199707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58387 (local address 127.0.97.62:41387)
0619 00:58:27.199927 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:27.199933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:27.200218 (+ 285us) server_negotiation.cc:407] Connection header received
0619 00:58:27.200406 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:27.200417 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:27.200463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:27.200564 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:27.201518 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:27.201523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:27.201530 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:27.201549 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:27.201583 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:27.201597 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:27.201722 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:27.201806 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 00:58:27.201862 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:58:27.203222 22420 log.cc:824] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:27.205857 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1: No bootstrap required, opened a new log
I20260619 00:58:27.213675 22420 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } }
I20260619 00:58:27.214084 22420 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:27.214190 22420 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d863b6ebfa499887e6af810de2afc1, State: Initialized, Role: FOLLOWER
I20260619 00:58:27.214748 22420 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } }
I20260619 00:58:27.214984 22420 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:27.215121 22420 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:27.215293 22420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:27.217689 22420 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } }
I20260619 00:58:27.218369 22420 leader_election.cc:302] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d863b6ebfa499887e6af810de2afc1; no voters:
I20260619 00:58:27.219004 22420 leader_election.cc:288] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:27.219152 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:27.219908 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [term 1 LEADER]: Becoming Leader. State: Replica: c1d863b6ebfa499887e6af810de2afc1, State: Running, Role: LEADER
I20260619 00:58:27.220438 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } }
I20260619 00:58:27.221211 22420 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:27.223259 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d863b6ebfa499887e6af810de2afc1. Latest consensus state: current_term: 1 leader_uuid: "c1d863b6ebfa499887e6af810de2afc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } } }
I20260619 00:58:27.224045 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:27.223937 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d863b6ebfa499887e6af810de2afc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d863b6ebfa499887e6af810de2afc1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41387 } } }
I20260619 00:58:27.224327 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:27.226325 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:27.231640 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:27.241091 22439 catalog_manager.cc:1347] Generated new cluster ID: ed15f964c01b4469ba2f83a5c45066c7
I20260619 00:58:27.241230 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:27.272362 22439 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:27.272907 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:27.284454 22439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d863b6ebfa499887e6af810de2afc1: Generated new TSK 0
I20260619 00:58:27.285365 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:27.293361 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22353
W20260619 00:58:27.294742 22415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37157
W20260619 00:58:27.295431 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:27.295583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22353
2026-06-19T00:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:27 dist-test-slave-r7bh krb5kdc[22451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:27 dist-test-slave-r7bh krb5kdc[22451](info): set up 2 sockets
Jun 19 00:58:27 dist-test-slave-r7bh krb5kdc[22451](info): commencing operation
krb5kdc: starting...
W20260619 00:58:28.405339 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:28 dist-test-slave-r7bh krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:28.667341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33625
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:29.014950 22464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:29.015246 22464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:29.015448 22464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:29.023412 22464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:29.023593 22464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:29.023681 22464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:29.023772 22464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:29.023840 22464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:29.034015 22464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:29.035400 22464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:29.036746 22464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:29.048612 22470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:29.048728 22469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:29.048700 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:29.049928 22464 server_base.cc:1034] running on GCE node
I20260619 00:58:29.050769 22464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:29.052199 22464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:29.053400 22464 hybrid_clock.cc:648] HybridClock initialized: now 1781830709053332 us; error 62 us; skew 500 ppm
I20260619 00:58:29.053730 22464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:29 dist-test-slave-r7bh krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:29.062043 22464 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:29.063795 22464 webserver.cc:460] Webserver started at http://127.0.97.62:41473/ using document root <none> and password file <none>
I20260619 00:58:29.064579 22464 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:29.064715 22464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:29.065063 22464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:29.068171 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "741e2ab3158e4d17801ff548d0927a08"
format_stamp: "Formatted at 2026-06-19 00:58:29 on dist-test-slave-r7bh"
server_key: "ba09ab8d6b90de09c5c7bc8194f18283"
server_key_iv: "f4ba0a8202eb537be9d8577e8a673522"
server_key_version: "encryptionkey@0"
I20260619 00:58:29.069116 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "741e2ab3158e4d17801ff548d0927a08"
format_stamp: "Formatted at 2026-06-19 00:58:29 on dist-test-slave-r7bh"
server_key: "ba09ab8d6b90de09c5c7bc8194f18283"
server_key_iv: "f4ba0a8202eb537be9d8577e8a673522"
server_key_version: "encryptionkey@0"
I20260619 00:58:29.074649 22464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:58:29.078251 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:29.079628 22464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:29.079820 22464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "741e2ab3158e4d17801ff548d0927a08"
format_stamp: "Formatted at 2026-06-19 00:58:29 on dist-test-slave-r7bh"
server_key: "ba09ab8d6b90de09c5c7bc8194f18283"
server_key_iv: "f4ba0a8202eb537be9d8577e8a673522"
server_key_version: "encryptionkey@0"
I20260619 00:58:29.080031 22464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:29.101408 22464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:29.106046 22464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:29.106362 22464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:29.125092 22464 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33625
I20260619 00:58:29.125113 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33625 every 8 connection(s)
I20260619 00:58:29.126571 22464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:29.130381 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:29.133438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22464
I20260619 00:58:29.133606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:29.134110 388 external_mini_cluster.cc:1428] Setting key 902381a741baf423efed96abbedba8a9
I20260619 00:58:29.143211 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: Bootstrap starting.
I20260619 00:58:29.147158 22531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:29.148572 22531 log.cc:824] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:58:29 dist-test-slave-r7bh krb5kdc[22451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:58:29.152216 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: No bootstrap required, opened a new log
I20260619 00:58:29.159471 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:29.136018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60415 (local address 127.0.97.62:33625)
0619 00:58:29.136933 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:29.136946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:29.136990 (+ 44us) server_negotiation.cc:407] Connection header received
0619 00:58:29.138680 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:29.138702 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:29.139108 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:29.139441 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:29.141178 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:29.142459 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:29.144272 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:29.145028 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:29.145178 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:29.150429 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:29.150439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:29.150459 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:29.150501 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:29.154692 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:29.155427 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:29.155442 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:29.155454 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:29.155676 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:29.156060 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:29.156065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:29.156069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:29.156803 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:29.157134 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:29.157523 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 00:58:29.158004 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":380,"threads_started":1}
I20260619 00:58:29.162966 22531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } }
I20260619 00:58:29.163395 22531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:29.163480 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:29.163516 22531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741e2ab3158e4d17801ff548d0927a08, State: Initialized, Role: FOLLOWER
I20260619 00:58:29.164171 22531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } }
I20260619 00:58:29.164424 22531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:29.164572 22531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:29.164755 22531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:29.167171 22531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } }
I20260619 00:58:29.167757 22531 leader_election.cc:302] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741e2ab3158e4d17801ff548d0927a08; no voters:
I20260619 00:58:29.168660 22531 leader_election.cc:288] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:29.168776 22540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:29.170771 22531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:29.175439 22540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [term 1 LEADER]: Becoming Leader. State: Replica: 741e2ab3158e4d17801ff548d0927a08, State: Running, Role: LEADER
I20260619 00:58:29.176134 22540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } }
I20260619 00:58:29.181239 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "741e2ab3158e4d17801ff548d0927a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } } }
I20260619 00:58:29.181655 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:29.182104 22543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741e2ab3158e4d17801ff548d0927a08. Latest consensus state: current_term: 1 leader_uuid: "741e2ab3158e4d17801ff548d0927a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741e2ab3158e4d17801ff548d0927a08" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33625 } } }
I20260619 00:58:29.182365 22543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:29.189816 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:29.168186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55357 (local address 127.0.97.62:33625)
0619 00:58:29.168399 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:29.168406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:29.168920 (+ 514us) server_negotiation.cc:407] Connection header received
0619 00:58:29.169173 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:29.169183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:29.169286 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:29.169394 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:29.172082 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:29.173705 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:29.176051 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:29.176941 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:29.177099 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:29.180531 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:29.180541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:29.180548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:29.180601 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:29.185426 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:29.186579 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:29.186588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:29.186594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:29.186778 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:29.187350 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:29.187359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:29.187364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:29.187926 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:29.188205 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:29.188968 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 00:58:29.189281 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 00:58:29.196779 22555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:29.196940 22555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:29.197113 22556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:29.198783 22556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:29.207396 22556 catalog_manager.cc:1347] Generated new cluster ID: a3b5c6b31c154407ba9e48c18d536ebb
I20260619 00:58:29.207523 22556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:29.225914 22556 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:29.227924 22556 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:29.241230 22556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741e2ab3158e4d17801ff548d0927a08: Generated new TSK 0
I20260619 00:58:29.242105 22556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:29.248728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22464
I20260619 00:58:29.249815 22526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:29.344107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22464
2026-06-19T00:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:29 dist-test-slave-r7bh krb5kdc[22563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:29 dist-test-slave-r7bh krb5kdc[22563](info): set up 2 sockets
Jun 19 00:58:29 dist-test-slave-r7bh krb5kdc[22563](info): commencing operation
krb5kdc: starting...
W20260619 00:58:30.513350 388 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:30 dist-test-slave-r7bh krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:30.785820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41301
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:31.121567 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:31.121848 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:31.122022 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:31.129184 22576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:31.129307 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:31.129395 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:31.129510 22576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:31.129571 22576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:31.139652 22576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:31.141072 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:31.142395 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:31.154773 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:31.154807 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:31.154592 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:31.155980 22576 server_base.cc:1034] running on GCE node
I20260619 00:58:31.156931 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:31.158468 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:31.159713 22576 hybrid_clock.cc:648] HybridClock initialized: now 1781830711159654 us; error 52 us; skew 500 ppm
I20260619 00:58:31.160111 22576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:31 dist-test-slave-r7bh krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:31.163421 22576 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:31.165067 22576 webserver.cc:460] Webserver started at http://127.0.97.62:45601/ using document root <none> and password file <none>
I20260619 00:58:31.165789 22576 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:31.165940 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:31.166250 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:31.169521 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4"
format_stamp: "Formatted at 2026-06-19 00:58:31 on dist-test-slave-r7bh"
server_key: "b2760a242c9176c99209c8b3aa3221c1"
server_key_iv: "127c7f4d042b07cf96dd3d3986114145"
server_key_version: "encryptionkey@0"
I20260619 00:58:31.170293 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4"
format_stamp: "Formatted at 2026-06-19 00:58:31 on dist-test-slave-r7bh"
server_key: "b2760a242c9176c99209c8b3aa3221c1"
server_key_iv: "127c7f4d042b07cf96dd3d3986114145"
server_key_version: "encryptionkey@0"
I20260619 00:58:31.175486 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 00:58:31.179288 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:31.180675 22576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:58:31.180919 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4"
format_stamp: "Formatted at 2026-06-19 00:58:31 on dist-test-slave-r7bh"
server_key: "b2760a242c9176c99209c8b3aa3221c1"
server_key_iv: "127c7f4d042b07cf96dd3d3986114145"
server_key_version: "encryptionkey@0"
I20260619 00:58:31.181149 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:31.205514 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:31.210443 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:31.210820 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:31.230341 22576 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41301
I20260619 00:58:31.230371 22642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41301 every 8 connection(s)
I20260619 00:58:31.231890 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:31.235906 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:31.242028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22576
I20260619 00:58:31.242236 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:31.242739 388 external_mini_cluster.cc:1428] Setting key 985c200e06bb5ce3b823e29980180beb
I20260619 00:58:31.247587 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4: Bootstrap starting.
I20260619 00:58:31.251085 22643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:31.252214 22643 log.cc:824] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:31.255625 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4: No bootstrap required, opened a new log
I20260619 00:58:31.257742 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:31.244868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41333 (local address 127.0.97.62:41301)
0619 00:58:31.245756 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:31.245770 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:31.245813 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:58:31.247643 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:31.247668 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:31.248175 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:31.248606 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:31.249262 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:31.250853 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:31.252483 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:31.253495 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:31.253659 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:31.254831 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:31.254841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:31.254860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:31.255567 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:31.255677 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:31.255720 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:31.255835 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:31.256060 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 00:58:31.256369 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":361,"threads_started":1}
I20260619 00:58:31.261907 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:31.264969 22643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } }
I20260619 00:58:31.265403 22643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:31.265512 22643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c31077acc2b4cb9b8bd4ecc30da97d4, State: Initialized, Role: FOLLOWER
I20260619 00:58:31.266206 22643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } }
I20260619 00:58:31.266546 22643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:31.266795 22643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:31.267002 22643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:31.269455 22643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } }
I20260619 00:58:31.269965 22643 leader_election.cc:302] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c31077acc2b4cb9b8bd4ecc30da97d4; no voters:
I20260619 00:58:31.270666 22643 leader_election.cc:288] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:31.270794 22653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:31.272449 22653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [term 1 LEADER]: Becoming Leader. State: Replica: 9c31077acc2b4cb9b8bd4ecc30da97d4, State: Running, Role: LEADER
I20260619 00:58:31.273124 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:31.266484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39393 (local address 127.0.97.62:41301)
0619 00:58:31.266683 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:31.266692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:31.267143 (+ 451us) server_negotiation.cc:407] Connection header received
0619 00:58:31.267378 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:31.267384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:31.267466 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:31.267583 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:31.268052 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:31.268795 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:31.270178 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:31.270692 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:31.270843 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:31.272199 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:31.272209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:31.272215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:31.272243 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:31.272293 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:31.272317 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:31.272414 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:31.272661 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 00:58:31.272757 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:58:31.273132 22653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } }
I20260619 00:58:31.273386 22643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:31.278548 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } } }
I20260619 00:58:31.279048 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:31.279699 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c31077acc2b4cb9b8bd4ecc30da97d4. Latest consensus state: current_term: 1 leader_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c31077acc2b4cb9b8bd4ecc30da97d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41301 } } }
I20260619 00:58:31.280046 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:31.281471 22664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:31.286351 22664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:31.303879 22664 catalog_manager.cc:1347] Generated new cluster ID: 58622a4e7c4c47c88b1f9ea68782c22e
I20260619 00:58:31.304072 22664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:31.320698 22664 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:31.322110 22664 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:31.343750 22664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c31077acc2b4cb9b8bd4ecc30da97d4: Generated new TSK 0
I20260619 00:58:31.345082 22664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:31.361176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22576
W20260619 00:58:31.362365 22638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41333
W20260619 00:58:31.363125 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:31.363292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22576
2026-06-19T00:58:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:31 dist-test-slave-r7bh krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:31 dist-test-slave-r7bh krb5kdc[22674](info): set up 2 sockets
Jun 19 00:58:31 dist-test-slave-r7bh krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260619 00:58:32.473398 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:32 dist-test-slave-r7bh krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:32.752030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43957
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:33.094601 22687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:33.094981 22687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:33.095170 22687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:33.102427 22687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:33.102630 22687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:33.102734 22687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:33.102843 22687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:33.102931 22687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:33.113737 22687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:33.115159 22687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:33.116525 22687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:33.128618 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:33.128713 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:33.128707 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:33.129886 22687 server_base.cc:1034] running on GCE node
I20260619 00:58:33.130723 22687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:33.132153 22687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:33.133355 22687 hybrid_clock.cc:648] HybridClock initialized: now 1781830713133274 us; error 69 us; skew 500 ppm
I20260619 00:58:33.133724 22687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:33 dist-test-slave-r7bh krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:33.137095 22687 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:33.138600 22687 webserver.cc:460] Webserver started at http://127.0.97.62:37195/ using document root <none> and password file <none>
I20260619 00:58:33.139281 22687 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:33.139389 22687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:33.139642 22687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:33.142669 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "9eae91b5909240c789403ffc288f1219"
format_stamp: "Formatted at 2026-06-19 00:58:33 on dist-test-slave-r7bh"
server_key: "beecbe881db6b35968973f5887a292c7"
server_key_iv: "86930ad058b598738a966769be48225a"
server_key_version: "encryptionkey@0"
I20260619 00:58:33.143404 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "9eae91b5909240c789403ffc288f1219"
format_stamp: "Formatted at 2026-06-19 00:58:33 on dist-test-slave-r7bh"
server_key: "beecbe881db6b35968973f5887a292c7"
server_key_iv: "86930ad058b598738a966769be48225a"
server_key_version: "encryptionkey@0"
I20260619 00:58:33.148718 22687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 00:58:33.152295 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:33.153795 22687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:58:33.153987 22687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "9eae91b5909240c789403ffc288f1219"
format_stamp: "Formatted at 2026-06-19 00:58:33 on dist-test-slave-r7bh"
server_key: "beecbe881db6b35968973f5887a292c7"
server_key_iv: "86930ad058b598738a966769be48225a"
server_key_version: "encryptionkey@0"
I20260619 00:58:33.154207 22687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:33.159914 22687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:33.164230 22687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:33.164533 22687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:33.183115 22687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43957
I20260619 00:58:33.183135 22753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43957 every 8 connection(s)
I20260619 00:58:33.184725 22687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:33.188736 22754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:33.189399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22687
I20260619 00:58:33.189603 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:33.190059 388 external_mini_cluster.cc:1428] Setting key 94c694a2379c997342bd1572ad88b8ed
I20260619 00:58:33.198571 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:33.191826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57327 (local address 127.0.97.62:43957)
0619 00:58:33.192736 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:33.192747 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:33.192790 (+ 43us) server_negotiation.cc:407] Connection header received
0619 00:58:33.195060 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:33.195089 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:33.195483 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:33.195856 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:33.196973 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:33.196980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:33.197006 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:33.197021 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:33.197081 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:33.197105 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:33.197276 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:33.197383 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 00:58:33.197593 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":395,"threads_started":1}
I20260619 00:58:33.202694 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:33.202838 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219: Bootstrap starting.
I20260619 00:58:33.206019 22754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:33.207350 22754 log.cc:824] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:33.209960 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:33.207139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37075 (local address 127.0.97.62:43957)
0619 00:58:33.207340 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:33.207349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:33.207894 (+ 545us) server_negotiation.cc:407] Connection header received
0619 00:58:33.208091 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:33.208095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:33.208146 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:33.208233 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:33.209299 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:33.209305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:33.209309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:58:33.209328 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:33.209372 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:58:33.209391 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:33.209523 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:33.209585 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 00:58:33.209642 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:58:33.210951 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219: No bootstrap required, opened a new log
I20260619 00:58:33.218432 22754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260619 00:58:33.218813 22754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:58:33.218894 22754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eae91b5909240c789403ffc288f1219, State: Initialized, Role: FOLLOWER
I20260619 00:58:33.219416 22754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260619 00:58:33.219611 22754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:58:33.219708 22754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:58:33.219830 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:33.222049 22754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260619 00:58:33.222500 22754 leader_election.cc:302] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eae91b5909240c789403ffc288f1219; no voters:
I20260619 00:58:33.223050 22754 leader_election.cc:288] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:58:33.223227 22764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:33.224670 22764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [term 1 LEADER]: Becoming Leader. State: Replica: 9eae91b5909240c789403ffc288f1219, State: Running, Role: LEADER
I20260619 00:58:33.225317 22754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:33.225232 22764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } }
I20260619 00:58:33.228185 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eae91b5909240c789403ffc288f1219. Latest consensus state: current_term: 1 leader_uuid: "9eae91b5909240c789403ffc288f1219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } } }
I20260619 00:58:33.228724 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:33.228266 22765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eae91b5909240c789403ffc288f1219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eae91b5909240c789403ffc288f1219" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43957 } } }
I20260619 00:58:33.229076 22765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:33.229810 22773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:33.235471 22773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:33.245895 22773 catalog_manager.cc:1347] Generated new cluster ID: 99927e326eee4fcea806e3ce43c0cfab
I20260619 00:58:33.246040 22773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:33.268110 22773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:33.268555 22773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:33.278028 22773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eae91b5909240c789403ffc288f1219: Generated new TSK 0
I20260619 00:58:33.278878 22773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:33.286003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22687
W20260619 00:58:33.287319 22749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57327
W20260619 00:58:33.288069 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:58:33.288225 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22687
2026-06-19T00:58:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1925 ms)
[----------] 18 tests from AuthTokenIssuingTest (35459 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:33 dist-test-slave-r7bh krb5kdc[22785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:33 dist-test-slave-r7bh krb5kdc[22785](info): set up 2 sockets
Jun 19 00:58:33 dist-test-slave-r7bh krb5kdc[22785](info): commencing operation
krb5kdc: starting...
W20260619 00:58:34.397425 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:34 dist-test-slave-r7bh krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:34.665171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43907
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:35.003600 22798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:35.003963 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:35.004136 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:35.004297 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:35.011989 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:35.012102 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:35.012202 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:35.012290 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:35.012389 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:35.022675 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:35.024214 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:35.025676 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:35.037946 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:35.037967 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:35.037899 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:35.039093 22798 server_base.cc:1034] running on GCE node
I20260619 00:58:35.039954 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:35.041540 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:35.042773 22798 hybrid_clock.cc:648] HybridClock initialized: now 1781830715042705 us; error 51 us; skew 500 ppm
I20260619 00:58:35.043135 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:35.049804 22798 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:35.051441 22798 webserver.cc:460] Webserver started at http://127.0.97.62:35081/ using document root <none> and password file <none>
I20260619 00:58:35.052194 22798 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:35.052332 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:35.052676 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:35.055939 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "4f721826cea547a594e0d20a4b2f4ba9"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "5a5a0a4e2d95c422bc6773467346c754"
server_key_iv: "1dd09776f61885e3dfbaf477b8e5ba22"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.056779 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f721826cea547a594e0d20a4b2f4ba9"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "5a5a0a4e2d95c422bc6773467346c754"
server_key_iv: "1dd09776f61885e3dfbaf477b8e5ba22"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.062177 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:58:35.065840 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:35.067209 22798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:35.067425 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "4f721826cea547a594e0d20a4b2f4ba9"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "5a5a0a4e2d95c422bc6773467346c754"
server_key_iv: "1dd09776f61885e3dfbaf477b8e5ba22"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.067655 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:35.082552 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:35.086776 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:35.087010 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:35.105506 22798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43907
I20260619 00:58:35.105512 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43907 every 8 connection(s)
I20260619 00:58:35.107023 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:35.110005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22798
I20260619 00:58:35.110244 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:35.111061 388 external_mini_cluster.cc:1428] Setting key 7070206407bfee08964d596c596ced7e
I20260619 00:58:35.111474 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:35.121942 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 00:58:35.145803 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.125287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.97.62:43907)
0619 00:58:35.126137 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:35.126155 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:35.126614 (+ 459us) server_negotiation.cc:407] Connection header received
0619 00:58:35.129233 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:35.129239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:35.129473 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:35.129687 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:35.131001 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.132195 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.133769 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.134321 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.134544 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.138168 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:35.138178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:35.138190 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:35.138220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:35.141356 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.142296 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.142303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.142305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.142451 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.142909 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.142912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.142914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.143419 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:35.143638 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:35.144285 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 00:58:35.144632 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
I20260619 00:58:35.145838 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.125288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43907 (local address 127.0.0.1:46424)
0619 00:58:35.126446 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.126480 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:35.128759 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:35.129709 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:35.129715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:35.130446 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:35.130783 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.130799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.132356 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.132363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.133632 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.133645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.134561 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.134567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.134733 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.135431 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:35.135454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:35.137845 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:35.141501 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.141516 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.141531 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.142142 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.142568 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.142576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.142582 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.142798 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.143578 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:35.143589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:35.143849 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 00:58:35.144140 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 00:58:35.144848 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":120,"thread_start_us":734,"threads_started":1}
I20260619 00:58:35.148885 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:35.150024 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.149512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36570)
0619 00:58:35.149723 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.149864 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 00:58:35.150257 22815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
I20260619 00:58:35.151777 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.151452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36576)
0619 00:58:35.151593 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.151666 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:58:35.151960 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42373: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 00:58:35.171947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42373
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:58:35.201261 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } attempt: 1
I20260619 00:58:35.202512 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.201971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36578)
0619 00:58:35.202208 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.202331 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 00:58:35.203593 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.203192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36590)
0619 00:58:35.203350 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.203436 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:58:35.203778 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42373: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
I20260619 00:58:35.272080 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } attempt: 2
I20260619 00:58:35.273458 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.272973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36592)
0619 00:58:35.273216 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.273314 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 00:58:35.274552 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.274216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36600)
0619 00:58:35.274343 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.274422 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 00:58:35.274871 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42373: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
I20260619 00:58:35.366134 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } attempt: 3
I20260619 00:58:35.367463 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.366971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36602)
0619 00:58:35.367218 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.367313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 00:58:35.368628 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.368235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36608)
0619 00:58:35.368394 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.368486 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:58:35.368854 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42373: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
W20260619 00:58:35.510691 22872 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:35.511122 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:35.511261 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:35.511412 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 00:58:35.512125 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } attempt: 4
I20260619 00:58:35.513600 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.512931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36616)
0619 00:58:35.513265 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.513384 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 00:58:35.514701 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.514324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36620)
0619 00:58:35.514484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.514582 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 00:58:35.514928 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42373: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42373: connect: Connection refused (error 111)
W20260619 00:58:35.519837 22872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:35.519980 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:35.520109 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:35.520197 22872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:35.520304 22872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:35.531008 22872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:35.532444 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:35.533869 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:35.547894 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:35.548709 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:35.548760 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:35.549150 22872 server_base.cc:1034] running on GCE node
I20260619 00:58:35.550010 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:35.551549 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:35.552881 22872 hybrid_clock.cc:648] HybridClock initialized: now 1781830715552733 us; error 64 us; skew 500 ppm
I20260619 00:58:35.553248 22872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:35.556926 22872 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 00:58:35.558609 22872 webserver.cc:460] Webserver started at http://127.0.97.61:36185/ using document root <none> and password file <none>
I20260619 00:58:35.559352 22872 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:35.559460 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:35.559755 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:35.563089 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data/instance:
uuid: "09aa69bf106043e3a96a6691aa06b035"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "586ebb470489785c2cb9de4cc61bbbad"
server_key_iv: "8ccd3702b620d449fa9b5db2e34d0f6c"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.563936 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal/instance:
uuid: "09aa69bf106043e3a96a6691aa06b035"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "586ebb470489785c2cb9de4cc61bbbad"
server_key_iv: "8ccd3702b620d449fa9b5db2e34d0f6c"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.569700 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 00:58:35.573962 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:35.575465 22872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260619 00:58:35.575686 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "09aa69bf106043e3a96a6691aa06b035"
format_stamp: "Formatted at 2026-06-19 00:58:35 on dist-test-slave-r7bh"
server_key: "586ebb470489785c2cb9de4cc61bbbad"
server_key_iv: "8ccd3702b620d449fa9b5db2e34d0f6c"
server_key_version: "encryptionkey@0"
I20260619 00:58:35.575932 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:35.596230 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:35.601140 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:35.601473 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:35.621603 22872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42373
I20260619 00:58:35.621632 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42373 every 8 connection(s)
I20260619 00:58:35.623169 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
I20260619 00:58:35.627063 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:35.627847 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22872
I20260619 00:58:35.628075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-1/wal/instance
I20260619 00:58:35.628530 388 external_mini_cluster.cc:1428] Setting key 7244916d2ea352760693f466ec319187
I20260619 00:58:35.633816 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 00:58:35.653931 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.636503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.0.97.62:43907)
0619 00:58:35.636737 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:35.636743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:35.637878 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 00:58:35.639750 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:35.639756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:35.639847 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:35.639962 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:35.641482 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.642171 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.643676 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.644088 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.644231 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.647764 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:35.647773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:35.647781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:35.647833 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:35.650787 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.651527 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.651532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.651534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.651682 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.652128 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.652133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.652136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.652433 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:35.652589 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:35.653373 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 00:58:35.653601 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 00:58:35.655351 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.636983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43907 (local address 127.0.0.1:46428)
0619 00:58:35.637703 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.637743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:35.639475 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:35.640191 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:35.640199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:35.640689 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:35.641316 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.641332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.642333 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.642346 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.643528 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.643540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.644313 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.644321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.644558 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.645338 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:35.645362 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:35.647477 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:35.650942 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.650952 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.650969 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.651388 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.651816 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.651821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.651824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.652011 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.652562 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:35.652570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:35.652986 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 00:58:35.653251 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 00:58:35.653773 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":292,"threads_started":1}
I20260619 00:58:35.657244 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:35.673960 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.657936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36630)
0619 00:58:35.658186 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.658206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:35.658319 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:35.659448 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:35.659456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:35.659998 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:35.660224 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.660239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.661715 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.661724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.663064 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.663076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.664337 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.664341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.664504 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.665164 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:35.665182 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:35.667544 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:35.671080 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.671087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.671094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.671693 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.672125 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.672132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.672138 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.672304 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.673159 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:35.673164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:35.673294 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 00:58:35.673431 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 00:58:35.673652 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":5}
I20260619 00:58:35.674532 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.658211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.0.97.61:42373)
0619 00:58:35.658981 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:35.658987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:35.659039 (+ 52us) server_negotiation.cc:407] Connection header received
0619 00:58:35.659084 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:35.659089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:35.659273 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:35.659387 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:35.660418 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.661561 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.663210 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.664180 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.664331 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.667790 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:35.667803 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:35.667817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:35.667853 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:35.670915 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.671857 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.671864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.671867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.671997 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.672427 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.672434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.672440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.672983 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:35.673200 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:35.673625 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 00:58:35.673970 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":619,"spinlock_wait_cycles":24960,"thread_start_us":468,"threads_started":1}
I20260619 00:58:35.677002 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:35.678373 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.677880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50282)
0619 00:58:35.678071 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.678194 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 00:58:35.678660 22891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.680447 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.680069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50284)
0619 00:58:35.680263 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.680343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 00:58:35.680696 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 00:58:35.688755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38585
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:58:35.730159 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 1
I20260619 00:58:35.731412 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.730905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50300)
0619 00:58:35.731104 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.731233 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 00:58:35.732427 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.732058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50306)
0619 00:58:35.732222 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.732294 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 00:58:35.732636 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.800869 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 2
I20260619 00:58:35.802063 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.801588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50316)
0619 00:58:35.801806 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.801912 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 00:58:35.803021 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.802700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50326)
0619 00:58:35.802843 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.802912 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 00:58:35.803218 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.814203 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } attempt: 5
Jun 19 00:58:35 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:35.828183 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.815080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36636)
0619 00:58:35.815305 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.815327 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:35.815477 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:35.815856 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:35.815864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:35.816285 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:35.816467 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.816476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.817571 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.817580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.818470 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:35.818482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.819229 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.819235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:35.819459 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.820165 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:35.820185 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:35.822355 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:35.825627 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.825633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.825637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.826040 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.826443 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:35.826448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:35.826452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:35.826600 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:35.827157 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:35.827162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:35.827342 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 00:58:35.827485 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 00:58:35.827804 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 00:58:35.828183 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.815118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.0.97.61:42373)
0619 00:58:35.815378 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:35.815385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:35.815408 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:58:35.815630 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:35.815634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:35.815700 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:35.815803 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:35.816650 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.817412 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.818624 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:35.819072 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:35.819210 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:35.822559 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:35.822565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:35.822568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:35.822601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:35.825470 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.826176 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.826188 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.826191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.826316 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:35.826712 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:35.826717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:35.826721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:35.827037 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:35.827197 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:35.827609 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 00:58:35.827837 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 00:58:35.829783 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:35.830976 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.830550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50328)
0619 00:58:35.830732 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.830826 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 00:58:35.831921 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.831603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50330)
0619 00:58:35.831739 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.831816 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 00:58:35.832149 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.883423 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 1
I20260619 00:58:35.884692 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.884105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50338)
0619 00:58:35.884362 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.884482 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 00:58:35.885933 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.885527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50350)
0619 00:58:35.885691 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.885782 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 00:58:35.886191 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.894443 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 3
I20260619 00:58:35.895593 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.895121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50352)
0619 00:58:35.895350 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.895440 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 00:58:35.896656 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.896250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50358)
0619 00:58:35.896469 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.896548 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 00:58:35.896873 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:35.954476 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 2
I20260619 00:58:35.955745 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.955140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50364)
0619 00:58:35.955395 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.955529 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 00:58:35.956842 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:35.956420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50376)
0619 00:58:35.956588 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:35.956673 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 00:58:35.957067 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
W20260619 00:58:36.032244 22946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:36.032651 22946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:36.032770 22946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:36.032941 22946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 00:58:36.040081 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 4
W20260619 00:58:36.040351 22946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:36.040480 22946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:36.040596 22946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:36.040685 22946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:36.040795 22946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:36.041414 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.040877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50378)
0619 00:58:36.041119 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.041225 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 00:58:36.042476 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.042127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50384)
0619 00:58:36.042274 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.042355 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 00:58:36.042707 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
I20260619 00:58:36.051612 22946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43907,127.0.97.61:42373,127.0.97.60:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:36.053072 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:36.054373 22946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 00:58:36.063287 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 3
I20260619 00:58:36.064530 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.063947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50390)
0619 00:58:36.064187 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.064302 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 00:58:36.065920 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.065368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50392)
0619 00:58:36.065665 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.065759 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260619 00:58:36.066035 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:36.066190 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38585: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38585: connect: Connection refused (error 111)
W20260619 00:58:36.066293 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:36.067612 22954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:36.067787 22946 server_base.cc:1034] running on GCE node
I20260619 00:58:36.068693 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:36.070163 22946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:36.071377 22946 hybrid_clock.cc:648] HybridClock initialized: now 1781830716071295 us; error 60 us; skew 500 ppm
I20260619 00:58:36.071734 22946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:36.074973 22946 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 00:58:36.076550 22946 webserver.cc:460] Webserver started at http://127.0.97.60:39667/ using document root <none> and password file <none>
I20260619 00:58:36.077339 22946 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:36.077479 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:36.077795 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:36.080960 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data/instance:
uuid: "5625c6d24aa7488d8bdf90fc8b68c194"
format_stamp: "Formatted at 2026-06-19 00:58:36 on dist-test-slave-r7bh"
server_key: "5d7bc4c48ed84c67ac369dfe48891abb"
server_key_iv: "ba6e17dbe13037ff30fdbb59d77f22ab"
server_key_version: "encryptionkey@0"
I20260619 00:58:36.081760 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal/instance:
uuid: "5625c6d24aa7488d8bdf90fc8b68c194"
format_stamp: "Formatted at 2026-06-19 00:58:36 on dist-test-slave-r7bh"
server_key: "5d7bc4c48ed84c67ac369dfe48891abb"
server_key_iv: "ba6e17dbe13037ff30fdbb59d77f22ab"
server_key_version: "encryptionkey@0"
I20260619 00:58:36.087177 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 00:58:36.090847 22961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:36.092293 22946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:58:36.092484 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "5625c6d24aa7488d8bdf90fc8b68c194"
format_stamp: "Formatted at 2026-06-19 00:58:36 on dist-test-slave-r7bh"
server_key: "5d7bc4c48ed84c67ac369dfe48891abb"
server_key_iv: "ba6e17dbe13037ff30fdbb59d77f22ab"
server_key_version: "encryptionkey@0"
I20260619 00:58:36.092702 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:36.109400 22946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:36.114182 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:36.114526 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:36.134079 22946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38585
I20260619 00:58:36.134099 23012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38585 every 8 connection(s)
I20260619 00:58:36.135589 22946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
I20260619 00:58:36.139810 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:36.145023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 22946
I20260619 00:58:36.145221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830351410501-388-0/minicluster-data/master-2/wal/instance
I20260619 00:58:36.145768 388 external_mini_cluster.cc:1428] Setting key 7751eeeea4f2664d861cb7d462a33091
I20260619 00:58:36.147934 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:58:36.166659 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.151217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.97.62:43907)
0619 00:58:36.151458 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.151465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.152094 (+ 629us) server_negotiation.cc:407] Connection header received
0619 00:58:36.152416 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.152421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.152490 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.152611 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.154160 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.154881 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.156437 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.156997 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.157146 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.160879 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.160885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.160888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.160921 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.163450 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.164274 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.164282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.164287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.164440 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.164902 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.164909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.164914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.165193 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.165365 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.166068 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.166294 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 00:58:36.167770 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.148055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.0.97.60:38585)
0619 00:58:36.148987 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.149022 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.149069 (+ 47us) server_negotiation.cc:407] Connection header received
0619 00:58:36.150881 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.150904 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.151468 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.151908 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.153480 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.154963 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.156236 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.156764 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.157036 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.161387 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.161395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.161434 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.161491 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.164461 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.165252 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.165264 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.165266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.165390 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.165778 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.165781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.165783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.166129 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.166443 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.166756 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.167014 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":637,"thread_start_us":382,"threads_started":1}
I20260619 00:58:36.167809 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.151148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43907 (local address 127.0.0.1:46436)
0619 00:58:36.151938 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.151983 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:36.152237 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:36.152683 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:36.152690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:36.153417 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:36.153973 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.153988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.155090 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.155094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.156266 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.156280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.157266 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.157277 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.157456 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.158064 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:36.158088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:36.160638 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:36.163600 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.163611 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.163624 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.164132 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.164560 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.164565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.164570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.164755 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.165333 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:36.165342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:36.165685 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 00:58:36.165978 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 00:58:36.166475 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":44,"thread_start_us":364,"threads_started":1}
I20260619 00:58:36.169823 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:58:36.183652 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.170416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42373 (local address 127.0.0.1:36638)
0619 00:58:36.170591 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.170607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:36.170724 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:36.171112 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:36.171119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:36.171576 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:36.171792 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.171806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.173022 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.173028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.174053 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.174065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.174815 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.174821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.175015 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.175577 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:36.175598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:36.177879 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:36.181213 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.181219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.181224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.181593 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.181981 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.181986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.181989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.182138 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.182663 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:36.182668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:36.182814 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 00:58:36.182977 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 00:58:36.183280 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 00:58:36.183681 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.170589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.0.97.61:42373)
0619 00:58:36.170784 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.170790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.170809 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:58:36.170871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.170875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.170935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.171066 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.171942 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.172866 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.174197 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.174667 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.174761 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.178085 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.178090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.178094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.178127 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.181085 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.181715 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.181722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.181725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.181864 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.182250 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.182256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.182259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.182528 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.182687 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.183084 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.183296 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:58:36.184716 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:36.186487 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.171482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.97.62:43907)
0619 00:58:36.171649 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.171653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.171670 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:58:36.171915 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.171920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.171993 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.172094 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.173426 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.174153 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.175730 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.176126 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.176218 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.180444 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.180449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.180452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.180492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.183152 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.184183 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.184188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.184192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.184373 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.184795 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.184800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.184804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.185194 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.185496 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.185792 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.186030 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:36.200711 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.185659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50406)
0619 00:58:36.185910 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.185937 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:36.186060 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:36.186353 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:36.186357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:36.186844 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:36.187057 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.187071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.188265 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.188270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.189784 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.189804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.190664 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.190668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.190912 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.191639 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:36.191665 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:36.194522 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:36.198000 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.198017 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.198023 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.198544 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.198920 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.198924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.198926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.199050 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.199721 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:36.199725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:36.199854 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 00:58:36.200017 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 00:58:36.200315 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":37}
I20260619 00:58:36.200932 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.185671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.0.97.60:38585)
0619 00:58:36.185835 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.185841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.186004 (+ 163us) server_negotiation.cc:407] Connection header received
0619 00:58:36.186180 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.186191 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.186243 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.186339 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.187224 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.188091 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.189943 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.190516 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.190649 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.194735 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.194742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.194749 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.194786 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.197855 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.198658 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.198664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.198667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.198794 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.199162 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.199169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.199174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.199577 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.199755 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.200122 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.200485 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":56}
I20260619 00:58:36.204438 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.188208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.0.97.61:42373)
0619 00:58:36.188358 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.188362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.188377 (+ 15us) server_negotiation.cc:407] Connection header received
0619 00:58:36.188519 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.188522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.188576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.188648 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.190224 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.191218 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.193172 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.193863 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.194003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.198528 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.198534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.198537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.198573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.201803 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.202514 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.202521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.202525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.202700 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.203075 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.203080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.203082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.203419 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.203699 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.203872 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.204109 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 00:58:36.207105 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: Bootstrap starting.
I20260619 00:58:36.209712 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:36.211134 23013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:36.212529 23013 log.cc:824] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:36.215790 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: No bootstrap required, opened a new log
I20260619 00:58:36.213296 22840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f721826cea547a594e0d20a4b2f4ba9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46444
I20260619 00:58:36.222980 23013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.223342 23013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5625c6d24aa7488d8bdf90fc8b68c194, State: Initialized, Role: FOLLOWER
I20260619 00:58:36.223865 23013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.224779 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.225292 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.226123 23013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 00:58:36.236074 23030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:36.236230 23030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:36.243407 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 4
I20260619 00:58:36.233911 22946 master_runner.cc:186] Error getting master registration for 127.0.97.62:43907: OK, instance_id { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" instance_seqno: 1781830715100183 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:58:36.244091 22946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:58:36.259023 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.244313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.97.60:38585)
0619 00:58:36.244935 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.244941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.244961 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:58:36.245024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.245031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.245107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.245207 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.246021 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.247198 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.248526 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.249155 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.249294 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.253215 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.253219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.253223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.253257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.256108 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.256852 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.256858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.256862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.257008 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.257419 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.257425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.257429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.257809 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.257997 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.258384 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.258620 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260619 00:58:36.259018 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.244212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50416)
0619 00:58:36.244490 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.244513 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:36.244648 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:36.245244 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:36.245249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:36.245680 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:36.245847 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.245857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.247379 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.247384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.248353 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.248366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.249327 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.249334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.249566 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.250252 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:36.250274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:36.253005 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:36.256272 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.256278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.256282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.256696 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.257131 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.257136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.257140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.257304 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.257961 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:36.257967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:36.258142 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 00:58:36.258290 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 00:58:36.258626 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 00:58:36.263684 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: Bootstrap starting.
I20260619 00:58:36.267902 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:36.269254 22865 log.cc:824] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:36.272392 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: No bootstrap required, opened a new log
I20260619 00:58:36.277667 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.278017 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f721826cea547a594e0d20a4b2f4ba9, State: Initialized, Role: FOLLOWER
I20260619 00:58:36.278553 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.279596 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.279973 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.280953 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:36.281016 22840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 00:58:36.281215 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:36.285498 22840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.288678 22988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f721826cea547a594e0d20a4b2f4ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5625c6d24aa7488d8bdf90fc8b68c194"
I20260619 00:58:36.289466 22988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:36.294363 22988 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f721826cea547a594e0d20a4b2f4ba9 in term 1.
I20260619 00:58:36.294919 22840 leader_election.cc:288] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [CANDIDATE]: Term 1 election: Requested vote from peers 09aa69bf106043e3a96a6691aa06b035 (127.0.97.61:42373), 5625c6d24aa7488d8bdf90fc8b68c194 (127.0.97.60:38585)
I20260619 00:58:36.295275 22817 leader_election.cc:302] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f721826cea547a594e0d20a4b2f4ba9, 5625c6d24aa7488d8bdf90fc8b68c194; no voters:
I20260619 00:58:36.295821 23033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:36.306756 23033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 4f721826cea547a594e0d20a4b2f4ba9, State: Running, Role: LEADER
I20260619 00:58:36.307652 23033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.316628 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f721826cea547a594e0d20a4b2f4ba9. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.317178 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:36.330525 23045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 00:58:36.339783 22819 tablet.cc:2369] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:58:36.341917 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } attempt: 5
I20260619 00:58:36.347275 23045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 00:58:36.368273 22967 tablet.cc:2369] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 00:58:36 dist-test-slave-r7bh krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:58:36.366019 22798 master_runner.cc:186] Error getting master registration for 127.0.97.61:42373: OK, instance_id { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" instance_seqno: 1781830715616035 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:58:36.384155 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.337162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.97.60:38585)
0619 00:58:36.337355 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.337361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.356692 (+ 19331us) server_negotiation.cc:407] Connection header received
0619 00:58:36.357233 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.357238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.357341 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.357419 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.359768 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.360934 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.364625 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.365314 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.365467 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.369442 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.369449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.369456 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.369502 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.374455 (+ 4953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.375789 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.375795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.375823 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.376098 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.376887 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.376892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.376895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.380365 (+ 3470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.380610 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.383472 (+ 2862us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.383735 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":68}
I20260619 00:58:36.386641 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.342687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38585 (local address 127.0.0.1:50442)
0619 00:58:36.343056 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:36.343078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:36.343208 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:36.358377 (+ 15169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:36.358386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:36.358926 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:36.359182 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.359197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.366361 (+ 7164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.366369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.367881 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:36.367898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.369741 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.369747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:36.369988 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.370912 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:36.370939 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:36.374335 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:36.383342 (+ 9007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.383350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.383355 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.383913 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.384654 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:36.384660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:36.384663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:36.384889 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:36.385556 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:36.385561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:36.385740 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 00:58:36.385901 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 00:58:36.386209 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":3}
I20260619 00:58:36.387122 22798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 00:58:36.387965 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.343452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.0.97.60:38585)
0619 00:58:36.356383 (+ 12931us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.356389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.356419 (+ 30us) server_negotiation.cc:407] Connection header received
0619 00:58:36.356494 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.356501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.356593 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.359383 (+ 2790us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.359658 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.366213 (+ 6555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.368873 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.369568 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.369718 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.374634 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.374641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.374646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.374693 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.383193 (+ 8500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.384064 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.384071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.384076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.384241 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.385046 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.385050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.385053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.385395 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.387108 (+ 1713us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.387256 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.387520 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":12753,"thread_start_us":12587,"threads_started":1}
I20260619 00:58:36.389472 22988 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [term 1 FOLLOWER]: Refusing update from remote peer 4f721826cea547a594e0d20a4b2f4ba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:58:36.390892 23033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:58:36.394703 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: Bootstrap starting.
I20260619 00:58:36.410940 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.357458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.0.97.61:42373)
0619 00:58:36.357653 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.357659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.364859 (+ 7200us) server_negotiation.cc:407] Connection header received
0619 00:58:36.366450 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.366457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.366539 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.366615 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.376979 (+ 10364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.378151 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.380099 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.380831 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.380982 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.385349 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.385356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.385361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.385403 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.389253 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.404688 (+ 15435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.404698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.404703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.404955 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.407165 (+ 2210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.407172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.407175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.407617 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.407881 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.410226 (+ 2345us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.410525 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":78}
I20260619 00:58:36.417145 22939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:36.424954 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f721826cea547a594e0d20a4b2f4ba9. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.425305 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.425479 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.337179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.0.97.62:43907)
0619 00:58:36.337355 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.337362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.355860 (+ 18498us) server_negotiation.cc:407] Connection header received
0619 00:58:36.361265 (+ 5405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.361270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.361359 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.361440 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.369974 (+ 8534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.371286 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.380196 (+ 8910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.381023 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.381152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.412292 (+ 31140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:36.412300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:36.412305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:36.412363 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:36.416361 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.420160 (+ 3799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.420168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.420174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.420378 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:36.421908 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:36.421915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:36.421918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:36.422356 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:36.422665 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.424357 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.425053 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 00:58:36.427053 22939 log.cc:824] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:36.431811 23057 mvcc.cc:205] Tried to move back new op lower bound from 7298378614314086400 to 7298378614004662272. Current Snapshot: MvccSnapshot[applied={T|T < 7298378614314086400}]
I20260619 00:58:36.436997 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.437345 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:36.438812 23058 mvcc.cc:205] Tried to move back new op lower bound from 7298378614314086400 to 7298378614004662272. Current Snapshot: MvccSnapshot[applied={T|T < 7298378614314086400}]
I20260619 00:58:36.441720 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: No bootstrap required, opened a new log
I20260619 00:58:36.449479 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.449800 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.451994 23045 catalog_manager.cc:1347] Generated new cluster ID: 40ccc35e879d4a8e8042aebbf821c3a0
I20260619 00:58:36.452109 23045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:36.459025 22939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.459774 22939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09aa69bf106043e3a96a6691aa06b035, State: Initialized, Role: FOLLOWER
I20260619 00:58:36.460687 22939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } }
I20260619 00:58:36.472508 22914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f721826cea547a594e0d20a4b2f4ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09aa69bf106043e3a96a6691aa06b035"
I20260619 00:58:36.473227 22939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:36.475304 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.475615 22913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:36.480003 22913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [term 1 FOLLOWER]: Refusing update from remote peer 4f721826cea547a594e0d20a4b2f4ba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:58:36.481002 23039 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:58:36.483906 22914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f721826cea547a594e0d20a4b2f4ba9 in term 1.
I20260619 00:58:36.485097 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.500025 23045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:36.502053 23045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:36.528426 23065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.528954 23065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.529225 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.529527 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:36.529544 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f721826cea547a594e0d20a4b2f4ba9. Latest consensus state: current_term: 1 leader_uuid: "4f721826cea547a594e0d20a4b2f4ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f721826cea547a594e0d20a4b2f4ba9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43907 } } peers { permanent_uuid: "09aa69bf106043e3a96a6691aa06b035" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42373 } } peers { permanent_uuid: "5625c6d24aa7488d8bdf90fc8b68c194" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38585 } } }
I20260619 00:58:36.529829 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:36.547047 23045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f721826cea547a594e0d20a4b2f4ba9: Generated new TSK 0
I20260619 00:58:36.552213 23045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:36.580713 23079 catalog_manager.cc:1259] Loaded cluster ID: 40ccc35e879d4a8e8042aebbf821c3a0
I20260619 00:58:36.580955 23079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: loading cluster ID for follower catalog manager: success
I20260619 00:58:36.585453 23079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: acquiring CA information for follower catalog manager: success
I20260619 00:58:36.588227 23079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09aa69bf106043e3a96a6691aa06b035: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 00:58:36.712997 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.703104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.97.60:38585)
0619 00:58:36.703306 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.703312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.709190 (+ 5878us) server_negotiation.cc:407] Connection header received
0619 00:58:36.710951 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.710958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.711051 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.711141 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.712712 (+ 1571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50450: BlockingRecv error: recv got EOF from 127.0.0.1:50450 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 00:58:36.714744 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.694490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.97.62:43907)
0619 00:58:36.694739 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.694746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.708646 (+ 13900us) server_negotiation.cc:407] Connection header received
0619 00:58:36.711360 (+ 2714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.711366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.711456 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.711544 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.714477 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 00:58:36.718919 22943 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.0.97.61:42373)
0619 00:58:36.710163 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.710170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.716316 (+ 6146us) server_negotiation.cc:407] Connection header received
0619 00:58:36.717499 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.717506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.717595 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.717666 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.718645 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36662: BlockingRecv error: recv got EOF from 127.0.0.1:36662 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:58:36.781368 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.752665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.97.61:42373)
0619 00:58:36.752917 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.752924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.763961 (+ 11037us) server_negotiation.cc:407] Connection header received
0619 00:58:36.764509 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.764517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.764728 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.764903 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:36.765615 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.772389 (+ 6774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.778141 (+ 5752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.778856 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.778976 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.779791 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:36.780520 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:36.780640 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.780912 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.781023 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 00:58:36.835222 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.826674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.0.97.60:38585)
0619 00:58:36.826875 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.826885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.833009 (+ 6124us) server_negotiation.cc:407] Connection header received
0619 00:58:36.833568 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.833574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.833724 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.833812 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.834940 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50456: BlockingRecv error: recv got EOF from 127.0.0.1:50456 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:58:36.892943 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.874608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.0.97.61:42373)
0619 00:58:36.874837 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.874844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.885590 (+ 10746us) server_negotiation.cc:407] Connection header received
0619 00:58:36.886345 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.886352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.886459 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.886532 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:36.887291 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.888449 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.890516 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.891202 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.891302 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.891946 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:36.892129 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:36.892200 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.892515 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.892604 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 00:58:36.914728 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.908033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.0.97.60:38585)
0619 00:58:36.908266 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.908275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.913234 (+ 4959us) server_negotiation.cc:407] Connection header received
0619 00:58:36.913488 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.913492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.913567 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.913684 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.914496 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50470: BlockingRecv error: recv got EOF from 127.0.0.1:50470 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:58:36.925207 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.919643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.0.97.61:42373)
0619 00:58:36.919884 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.919893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.920382 (+ 489us) server_negotiation.cc:407] Connection header received
0619 00:58:36.920596 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.920602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.920684 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.920785 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:36.921233 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.922018 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.923341 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.923796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.923949 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.924339 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:36.924506 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:36.924598 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.924741 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.924855 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 00:58:36.937014 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.934764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.60:38585)
0619 00:58:36.934999 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.935008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.935549 (+ 541us) server_negotiation.cc:407] Connection header received
0619 00:58:36.935773 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.935778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.935875 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.935992 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.936742 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50478: BlockingRecv error: recv got EOF from 127.0.0.1:50478 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:58:36.958112 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.951799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.0.97.61:42373)
0619 00:58:36.952134 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.952145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.952803 (+ 658us) server_negotiation.cc:407] Connection header received
0619 00:58:36.953095 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.953101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.953211 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.953323 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:36.953809 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.954641 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.956149 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.956671 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.956856 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.957287 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:36.957406 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:36.957530 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.957682 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.957773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260619 00:58:36.976558 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:36.974490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.0.97.60:38585)
0619 00:58:36.974747 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.974753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.975056 (+ 303us) server_negotiation.cc:407] Connection header received
0619 00:58:36.975293 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.975300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.975398 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.975519 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:36.976344 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50480: BlockingRecv error: recv got EOF from 127.0.0.1:50480 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 00:58:36.997068 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:36.990799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.0.97.61:42373)
0619 00:58:36.991065 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:36.991075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:36.991662 (+ 587us) server_negotiation.cc:407] Connection header received
0619 00:58:36.991917 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:36.991922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:36.991996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:36.992109 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:36.992553 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.993508 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.994841 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:36.995426 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:36.995577 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:36.996074 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:36.996262 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:36.996380 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:36.996569 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 00:58:36.996656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 00:58:37.005311 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:37.003212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.97.60:38585)
0619 00:58:37.003452 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.003459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.003851 (+ 392us) server_negotiation.cc:407] Connection header received
0619 00:58:37.004032 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.004038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.004140 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.004268 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:37.005070 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50482: BlockingRecv error: recv got EOF from 127.0.0.1:50482 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:58:37.033492 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.026066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.0.97.61:42373)
0619 00:58:37.026507 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.026527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.026568 (+ 41us) server_negotiation.cc:407] Connection header received
0619 00:58:37.027066 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.027074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.027203 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.027542 (+ 339us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.028201 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.029519 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.031123 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.031807 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.031976 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.032556 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.032717 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.032873 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.033087 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.033196 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
W20260619 00:58:37.042820 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:37.040218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.97.60:38585)
0619 00:58:37.040472 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.040480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.041036 (+ 556us) server_negotiation.cc:407] Connection header received
0619 00:58:37.041262 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.041268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.041379 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.041536 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:37.042564 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50484: BlockingRecv error: recv got EOF from 127.0.0.1:50484 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:58:37.088075 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.081895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.0.97.61:42373)
0619 00:58:37.082156 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.082163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.082492 (+ 329us) server_negotiation.cc:407] Connection header received
0619 00:58:37.082781 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.082788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.082888 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.083015 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.083386 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.084575 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.085985 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.086669 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.086813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.087217 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.087362 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.087475 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.087622 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.087711 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 00:58:37.096521 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:37.094459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.0.97.60:38585)
0619 00:58:37.094700 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.094707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.095017 (+ 310us) server_negotiation.cc:407] Connection header received
0619 00:58:37.095251 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.095258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.095364 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.095486 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:37.096268 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50498: BlockingRecv error: recv got EOF from 127.0.0.1:50498 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:58:37.169886 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.164682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.0.97.61:42373)
0619 00:58:37.164925 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.164930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.165406 (+ 476us) server_negotiation.cc:407] Connection header received
0619 00:58:37.165569 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.165574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.165645 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.165736 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.166131 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.166868 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.168178 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.168635 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.168733 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.169167 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.169295 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.169364 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.169591 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.169662 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 00:58:37.178061 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:37.176139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.0.97.60:38585)
0619 00:58:37.176386 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.176393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.176632 (+ 239us) server_negotiation.cc:407] Connection header received
0619 00:58:37.176803 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.176844 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.176938 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.177035 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:37.177824 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50508: BlockingRecv error: recv got EOF from 127.0.0.1:50508 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 00:58:37.238246 23030 catalog_manager.cc:1259] Loaded cluster ID: 40ccc35e879d4a8e8042aebbf821c3a0
I20260619 00:58:37.238354 23030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: loading cluster ID for follower catalog manager: success
I20260619 00:58:37.240913 23030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: acquiring CA information for follower catalog manager: success
I20260619 00:58:37.242888 23030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5625c6d24aa7488d8bdf90fc8b68c194: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:58:37.316144 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.310273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.0.97.61:42373)
0619 00:58:37.310463 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.310468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.310843 (+ 375us) server_negotiation.cc:407] Connection header received
0619 00:58:37.311016 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.311020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.311105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.311201 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.311675 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.312430 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.314234 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.314696 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.314801 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.315340 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.315473 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.315555 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.315814 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.315892 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:58:37.328274 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.322100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.0.97.60:38585)
0619 00:58:37.322332 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.322338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.322648 (+ 310us) server_negotiation.cc:407] Connection header received
0619 00:58:37.322869 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.322873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.323070 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.323204 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.323696 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.324428 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.325850 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.326352 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.326491 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.327048 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.327643 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.327834 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.327885 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.327984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 00:58:37.342937 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.335106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.0.97.62:43907)
0619 00:58:37.336280 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.336286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.336305 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:58:37.336416 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.336420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.336580 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.336767 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.337327 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.338286 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.340466 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.340934 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.341084 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.341588 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.342166 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.342280 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.342510 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.342600 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":930,"threads_started":1}
I20260619 00:58:37.343990 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.335284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.97.60:38585)
0619 00:58:37.335540 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.335549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.337514 (+ 1965us) server_negotiation.cc:407] Connection header received
0619 00:58:37.337855 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.337865 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.338000 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.338082 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.339096 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.340004 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.341973 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.342625 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.342743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.343282 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.343426 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.343522 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.343659 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.343740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 00:58:37.346078 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:37.337712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.0.97.61:42373)
0619 00:58:37.337879 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:37.337885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:37.339103 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 00:58:37.339383 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:37.339391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:37.339482 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:37.339579 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:37.340237 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.341394 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.343396 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:37.344115 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:37.344217 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:37.345297 (+ 1080us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:37.345461 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:37.345540 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:37.345723 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 00:58:37.345815 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:58:37.349980 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22798
I20260619 00:58:37.351125 22860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:37.462770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22798
W20260619 00:58:37.484344 22962 connection.cc:535] server connection from 127.0.0.1:50416 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 00:58:37.484736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22872
I20260619 00:58:37.485944 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:37.596429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22872
I20260619 00:58:37.617478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 22946
I20260619 00:58:37.618635 23008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:37.727277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 22946
2026-06-19T00:58:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4444 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:58:37 dist-test-slave-r7bh krb5kdc[23193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:58:37 dist-test-slave-r7bh krb5kdc[23193](info): set up 2 sockets
Jun 19 00:58:37 dist-test-slave-r7bh krb5kdc[23193](info): commencing operation
krb5kdc: starting...
W20260619 00:58:38.845911 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/test-user.keytab.
Jun 19 00:58:39 dist-test-slave-r7bh krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:58:39.124491 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34697
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:58:39.467983 23206 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:39.468400 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:39.468503 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:39.468639 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:39.476179 23206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:39.476300 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:39.476363 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:39.476477 23206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:39.476590 23206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:39.487104 23206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:39.488538 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:39.489975 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:39.501770 23211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:39.501959 23212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:39.502275 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:39.503113 23206 server_base.cc:1034] running on GCE node
I20260619 00:58:39.503916 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:39.505512 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:39.506788 23206 hybrid_clock.cc:648] HybridClock initialized: now 1781830719506724 us; error 54 us; skew 500 ppm
I20260619 00:58:39.507289 23206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:39 dist-test-slave-r7bh krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:39.510866 23206 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:58:39.512472 23206 webserver.cc:460] Webserver started at http://127.0.97.62:45977/ using document root <none> and password file <none>
I20260619 00:58:39.513288 23206 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:39.513384 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:39.513691 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:39.516863 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data/instance:
uuid: "2cbbbf6da815447d823dafb5d330e0b4"
format_stamp: "Formatted at 2026-06-19 00:58:39 on dist-test-slave-r7bh"
server_key: "aa17b8ef5f0c9b0891695c9f379df7de"
server_key_iv: "db5de742b5e0427fcc55137600971434"
server_key_version: "encryptionkey@0"
I20260619 00:58:39.517695 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance:
uuid: "2cbbbf6da815447d823dafb5d330e0b4"
format_stamp: "Formatted at 2026-06-19 00:58:39 on dist-test-slave-r7bh"
server_key: "aa17b8ef5f0c9b0891695c9f379df7de"
server_key_iv: "db5de742b5e0427fcc55137600971434"
server_key_version: "encryptionkey@0"
I20260619 00:58:39.523356 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 00:58:39.527297 23221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:39.528925 23206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:58:39.529145 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal
uuid: "2cbbbf6da815447d823dafb5d330e0b4"
format_stamp: "Formatted at 2026-06-19 00:58:39 on dist-test-slave-r7bh"
server_key: "aa17b8ef5f0c9b0891695c9f379df7de"
server_key_iv: "db5de742b5e0427fcc55137600971434"
server_key_version: "encryptionkey@0"
I20260619 00:58:39.529397 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:39.546623 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:39.551476 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:39.551796 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:39.572412 23206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34697
I20260619 00:58:39.572463 23272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34697 every 8 connection(s)
I20260619 00:58:39.573858 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:58:39.577718 23273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:39.581782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 23206
I20260619 00:58:39.581979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-0/wal/instance
I20260619 00:58:39.582518 388 external_mini_cluster.cc:1428] Setting key 803d92c57526b122bb4376b51db7ddf4
I20260619 00:58:39.586127 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:39 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 00:58:39.614313 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.590390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.0.1:51558)
0619 00:58:39.591759 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.591805 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:39.594015 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:39.595244 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:39.595255 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:39.596167 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:39.596714 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:39.596743 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:39.598416 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:39.598424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:39.600415 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:39.600434 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:39.601642 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:39.601652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:39.601885 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:39.602633 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:39.602669 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:39.605950 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:39.609564 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:39.609579 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:39.609593 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:39.610216 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:39.610678 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:39.610685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:39.610689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:39.610914 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:39.612006 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:39.612013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:39.612243 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 00:58:39.612481 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 00:58:39.613052 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"spinlock_wait_cycles":35584,"thread_start_us":838,"threads_started":1}
I20260619 00:58:39.614360 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.590390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.62:34697)
0619 00:58:39.591342 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:39.591362 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:39.591966 (+ 604us) server_negotiation.cc:407] Connection header received
0619 00:58:39.594527 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:39.594534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:39.594929 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:39.595128 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:39.596994 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:39.598236 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:39.600626 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:39.601341 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:39.601492 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:39.606252 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:39.606261 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:39.606276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:39.606311 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:39.609394 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:39.610387 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:39.610394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:39.610396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:39.610551 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:39.611127 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:39.611135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:39.611139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:39.611781 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:39.611994 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:39.612669 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 00:58:39.613166 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":249,"server-negotiator.queue_time_us":668,"thread_start_us":411,"threads_started":1}
I20260619 00:58:39.617727 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:39.618984 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.618450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50240)
0619 00:58:39.618735 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.618844 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260619 00:58:39.619259 23225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
I20260619 00:58:39.620966 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.620541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50244)
0619 00:58:39.620691 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.620771 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:58:39.621167 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40641: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 00:58:39.647917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40641
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:58:39.670589 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } attempt: 1
I20260619 00:58:39.672052 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.671414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50248)
0619 00:58:39.671683 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.671829 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 00:58:39.673393 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.672901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50254)
0619 00:58:39.673089 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.673196 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:58:39.673678 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40641: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
I20260619 00:58:39.741969 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } attempt: 2
I20260619 00:58:39.743321 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.742774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50264)
0619 00:58:39.743042 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.743142 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 00:58:39.744518 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.744086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50268)
0619 00:58:39.744266 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.744370 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 00:58:39.744748 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40641: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
I20260619 00:58:39.836107 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } attempt: 3
I20260619 00:58:39.837270 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.836782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50282)
0619 00:58:39.837027 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.837128 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 00:58:39.838336 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.837979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50286)
0619 00:58:39.838133 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.838202 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 00:58:39.838519 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40641: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
I20260619 00:58:39.981783 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } attempt: 4
I20260619 00:58:39.983160 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.982583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50298)
0619 00:58:39.982922 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.983023 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 00:58:39.984263 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:39.983888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50310)
0619 00:58:39.984045 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:39.984138 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 00:58:39.984475 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40641: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40641: connect: Connection refused (error 111)
W20260619 00:58:39.994249 23280 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:39.994678 23280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:39.994807 23280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:39.994962 23280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:40.002667 23280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:40.002817 23280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:40.002950 23280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:40.003047 23280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:40.003161 23280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:40.013485 23280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:40.014925 23280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:40.016311 23280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:40.028688 23285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:40.028729 23288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:40.028882 23286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:40.030028 23280 server_base.cc:1034] running on GCE node
I20260619 00:58:40.030921 23280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:40.032565 23280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:40.033937 23280 hybrid_clock.cc:648] HybridClock initialized: now 1781830720033739 us; error 117 us; skew 500 ppm
I20260619 00:58:40.034366 23280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:40.037927 23280 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 00:58:40.039587 23280 webserver.cc:460] Webserver started at http://127.0.97.61:37979/ using document root <none> and password file <none>
I20260619 00:58:40.040335 23280 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:40.040468 23280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:40.040802 23280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:40.044104 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data/instance:
uuid: "cf54dde6795c46b087e951f0f6979270"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "8a649503c26005737494562d8c06d808"
server_key_iv: "f3a2cef6a57b569809e46e9d81908c52"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.045054 23280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal/instance:
uuid: "cf54dde6795c46b087e951f0f6979270"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "8a649503c26005737494562d8c06d808"
server_key_iv: "f3a2cef6a57b569809e46e9d81908c52"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.050639 23280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:58:40.054785 23295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:40.056627 23280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 00:58:40.056924 23280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal
uuid: "cf54dde6795c46b087e951f0f6979270"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "8a649503c26005737494562d8c06d808"
server_key_iv: "f3a2cef6a57b569809e46e9d81908c52"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.057199 23280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:40.077162 23280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:40.081321 23280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:40.081607 23280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:40.101135 23280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40641
I20260619 00:58:40.101143 23346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40641 every 8 connection(s)
I20260619 00:58:40.102551 23280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/data/info.pb
I20260619 00:58:40.105098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 23280
I20260619 00:58:40.105299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-1/wal/instance
I20260619 00:58:40.105727 388 external_mini_cluster.cc:1428] Setting key a04ebf29e84a2f595ebe7c07a62cf222
I20260619 00:58:40.106597 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:40.113399 23347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 00:58:40.136178 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.116263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.0.97.62:34697)
0619 00:58:40.116853 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.116862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.117910 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 00:58:40.119848 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.119856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.119963 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.120104 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.121776 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.123066 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.124722 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.125229 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.125373 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.129304 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.129310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.129313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.129404 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.132368 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.133478 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.133482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.133485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.133611 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.134143 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.134148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.134150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.134606 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.134733 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.135718 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.135907 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":280,"threads_started":1}
I20260619 00:58:40.137827 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.116891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.0.1:51572)
0619 00:58:40.117708 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.117764 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.119571 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.120308 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.120316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.120970 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.121534 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.121552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.123250 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.123256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.124505 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.124517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.125494 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.125501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.125736 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.126457 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.126484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.129028 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.132553 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.132566 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.132589 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.133334 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.133730 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.133738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.133745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.134009 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.134745 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.134756 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.135302 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.135623 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.136221 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":356,"threads_started":1}
I20260619 00:58:40.139685 23347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:40.158588 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.140362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50322)
0619 00:58:40.140555 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.140573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.140673 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.143241 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.143245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.143625 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.143767 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.143775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.145905 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.145909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.146747 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.146757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.147992 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.147996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.148132 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.148768 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.148787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.150890 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.155814 (+ 4924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.155818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.155820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.156186 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.156713 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.156717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.156719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.156955 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.157862 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.157866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.157993 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.158104 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.158320 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":3}
I20260619 00:58:40.159142 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.140638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.0.97.61:40641)
0619 00:58:40.142704 (+ 2066us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.142715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.142749 (+ 34us) server_negotiation.cc:407] Connection header received
0619 00:58:40.142817 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.142827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.143079 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.143231 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.143939 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.145736 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.146896 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.147845 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.147989 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.151149 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.151168 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.151190 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.151245 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.155638 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.156364 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.156377 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.156382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.156559 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.157076 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.157085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.157092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.157662 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.157912 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.158329 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.158636 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":1716,"threads_started":1}
I20260619 00:58:40.161993 23347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:40.163206 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.162727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40084)
0619 00:58:40.162945 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.163068 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 00:58:40.163618 23296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.165345 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.165012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40088)
0619 00:58:40.165163 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.165243 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 00:58:40.165617 23347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 00:58:40.168098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6V91bC/build/asan/bin/kudu
/tmp/dist-test-task6V91bC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33837
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 00:58:40.214955 23347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 1
I20260619 00:58:40.216238 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.215757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40092)
0619 00:58:40.215979 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.216067 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 00:58:40.217401 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.217028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40094)
0619 00:58:40.217206 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.217288 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 00:58:40.217612 23347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.283715 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } attempt: 5
I20260619 00:58:40.285818 23347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 2
I20260619 00:58:40.286790 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.286369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40104)
0619 00:58:40.286537 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.286639 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 00:58:40.287793 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.287465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40112)
0619 00:58:40.287607 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.287678 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 00:58:40.288050 23347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:40.298502 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.284712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.0.97.61:40641)
0619 00:58:40.284928 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.284934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.284958 (+ 24us) server_negotiation.cc:407] Connection header received
0619 00:58:40.285049 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.285055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.285155 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.285279 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.286202 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.287452 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.289227 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.289662 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.289762 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.293241 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.293247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.293250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.293285 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.295853 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.296530 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.296534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.296536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.296655 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.297082 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.297086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.297088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.297370 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.297531 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.297919 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.298159 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 00:58:40.298521 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.284508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50334)
0619 00:58:40.284733 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.284756 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.284943 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.285301 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.285304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.285724 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.286017 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.286029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.287654 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.287660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.289073 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.289089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.289829 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.289836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.290020 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.290626 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.290650 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.293033 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.296001 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.296007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.296011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.296390 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.296771 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.296776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.296779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.296960 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.297491 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.297496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.297664 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.297808 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.298148 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 00:58:40.299775 23273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:40.300911 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.300408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40126)
0619 00:58:40.300576 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.300690 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 00:58:40.301846 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.301519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40130)
0619 00:58:40.301661 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.301744 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 00:58:40.302057 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.353345 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 1
I20260619 00:58:40.354671 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.354055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40144)
0619 00:58:40.354323 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.354449 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 00:58:40.355834 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.355465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40148)
0619 00:58:40.355623 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.355708 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:58:40.356065 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.379316 23347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 3
I20260619 00:58:40.380535 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.380033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40156)
0619 00:58:40.380295 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.380381 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 00:58:40.381642 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.381229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40168)
0619 00:58:40.381451 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.381522 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 00:58:40.381896 23347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.424312 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 2
I20260619 00:58:40.425653 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.425124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40184)
0619 00:58:40.425383 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.425493 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 00:58:40.426792 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.426430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40198)
0619 00:58:40.426599 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.426668 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 00:58:40.427021 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
W20260619 00:58:40.515455 23355 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 00:58:40.515863 23355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:58:40.515990 23355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:58:40.516130 23355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:58:40.523716 23355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:58:40.523873 23355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:58:40.524006 23355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:58:40.524094 23355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:58:40.524205 23355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:58:40.525116 23347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 4
I20260619 00:58:40.526381 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.525848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40204)
0619 00:58:40.526108 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.526200 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 00:58:40.527448 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.527098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40218)
0619 00:58:40.527254 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.527334 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 00:58:40.527700 23347 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.533272 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 3
I20260619 00:58:40.534459 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.533945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40234)
0619 00:58:40.534153 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.534292 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 00:58:40.535295 23355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34697,127.0.97.61:40641,127.0.97.60:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:58:40.535640 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.535196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40248)
0619 00:58:40.535416 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.535489 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260619 00:58:40.535845 23273 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33837: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33837: connect: Connection refused (error 111)
I20260619 00:58:40.536856 23355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:58:40.538231 23355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:58:40.550884 23361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:40.550992 23360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:58:40.551533 23363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:58:40.551935 23355 server_base.cc:1034] running on GCE node
I20260619 00:58:40.552778 23355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:58:40.554466 23355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:58:40.555764 23355 hybrid_clock.cc:648] HybridClock initialized: now 1781830720555683 us; error 43 us; skew 500 ppm
I20260619 00:58:40.556208 23355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:58:40.560559 23355 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 00:58:40.562882 23355 webserver.cc:460] Webserver started at http://127.0.97.60:45033/ using document root <none> and password file <none>
I20260619 00:58:40.563684 23355 fs_manager.cc:359] Metadata directory not provided
I20260619 00:58:40.563833 23355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:58:40.564173 23355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:58:40.567596 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data/instance:
uuid: "75627595593f47158c27863188923a37"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "6de06cfb48de79b0db80e629f4a0e4d2"
server_key_iv: "d740b24bef221f01a2f5c8d32a1b1b88"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.568517 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal/instance:
uuid: "75627595593f47158c27863188923a37"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "6de06cfb48de79b0db80e629f4a0e4d2"
server_key_iv: "d740b24bef221f01a2f5c8d32a1b1b88"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.574561 23355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 00:58:40.578630 23370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:58:40.580199 23355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:58:40.580492 23355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data,/tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal
uuid: "75627595593f47158c27863188923a37"
format_stamp: "Formatted at 2026-06-19 00:58:40 on dist-test-slave-r7bh"
server_key: "6de06cfb48de79b0db80e629f4a0e4d2"
server_key_iv: "d740b24bef221f01a2f5c8d32a1b1b88"
server_key_version: "encryptionkey@0"
I20260619 00:58:40.580724 23355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:58:40.616042 23355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:58:40.620482 23355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:58:40.620798 23355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:58:40.640084 23355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33837
I20260619 00:58:40.640110 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33837 every 8 connection(s)
I20260619 00:58:40.641577 23355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/data/info.pb
I20260619 00:58:40.645370 23422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:58:40.647382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6V91bC/build/asan/bin/kudu as pid 23355
I20260619 00:58:40.647578 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6V91bC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830351410501-388-0/minicluster-data/master-2/wal/instance
I20260619 00:58:40.648104 388 external_mini_cluster.cc:1428] Setting key 47ca46d162f4539af1aacc03de8acef8
I20260619 00:58:40.655841 23422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:58:40.672461 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.649944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.97.60:33837)
0619 00:58:40.650759 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.650775 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.650826 (+ 51us) server_negotiation.cc:407] Connection header received
0619 00:58:40.652526 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.652557 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.653165 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.653657 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.654943 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.657010 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.658360 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.659262 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.660031 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.664015 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.664023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.664035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.664078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.667630 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.668336 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.668351 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.668358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.668508 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.668997 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.669004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.669009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.669756 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.670079 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.670582 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.671055 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":330,"threads_started":1}
I20260619 00:58:40.674316 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.658882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34697 (local address 127.0.0.1:51582)
0619 00:58:40.659809 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.659851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.660010 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.660528 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.660533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.661188 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.661581 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.661591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.662920 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.662926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.664060 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.664072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.664862 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.664869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.665062 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.665667 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.665684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.668021 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.671675 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.671685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.671690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.672093 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.672503 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.672506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.672508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.672669 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.673299 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.673306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.673531 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.673737 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.674023 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":556,"threads_started":1}
I20260619 00:58:40.674532 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.658844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.0.97.62:34697)
0619 00:58:40.659372 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.659379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.660114 (+ 735us) server_negotiation.cc:407] Connection header received
0619 00:58:40.660207 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.660215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.660313 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.660441 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.661739 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.662762 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.664223 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.664628 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.664773 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.668213 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.668217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.668221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.668263 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.671528 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.672214 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.672220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.672223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.672387 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.672858 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.672866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.672872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.673151 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.673274 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.673865 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.674127 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260619 00:58:40.675830 23422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:58:40.689184 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.676539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40641 (local address 127.0.0.1:50338)
0619 00:58:40.676675 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.676693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.676793 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.677197 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.677202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.677617 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.677751 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.677759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.678720 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.678726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.679643 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.679653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.680359 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.680365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.680558 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.681137 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.681154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.683183 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.686657 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.686663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.686667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.687051 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.687480 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.687486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.687489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.687636 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.688159 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.688164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.688326 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.688468 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.688775 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 00:58:40.689225 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.676690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.0.97.61:40641)
0619 00:58:40.676929 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.676935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.676952 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:58:40.677009 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.677014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.677070 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.677177 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.677898 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.678573 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.679805 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.680205 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.680345 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.683388 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.683393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.683396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.683430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.686508 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.687182 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.687188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.687192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.687363 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.687753 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.687760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.687763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.688032 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.688202 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.688566 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.688790 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 00:58:40.690182 23422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 00:58:40.690450 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.675957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.0.97.62:34697)
0619 00:58:40.676399 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.676403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.676423 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:58:40.676468 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.676471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.676561 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.676652 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.678002 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.678681 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.679884 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.680304 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.680435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.684698 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.684703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.684706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.684738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.687478 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.688172 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.688178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.688182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.688311 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.688686 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.688692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.688696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.689029 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.689776 (+ 747us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.689916 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.690169 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 00:58:40.704641 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.691096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40260)
0619 00:58:40.691376 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.691395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.691500 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.691808 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.691812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.692289 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.692480 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.692493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.693464 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.693469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.695006 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.695023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.695754 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.695759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.695973 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.696591 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.696609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.698946 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.702108 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.702115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.702119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.702476 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.702900 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.702904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.702907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.703044 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.703712 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.703715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.703863 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.703985 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.704243 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260619 00:58:40.704900 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.691108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.97.60:33837)
0619 00:58:40.691276 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.691281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.691473 (+ 192us) server_negotiation.cc:407] Connection header received
0619 00:58:40.691597 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.691602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.691674 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.691747 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.692588 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.693340 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.695156 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.695587 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.695738 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.699112 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.699117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.699122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.699158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.701983 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.702590 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.702597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.702601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.702764 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.703163 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.703168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.703172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.703543 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.703713 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.704106 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.704445 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":52}
I20260619 00:58:40.708917 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.692259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.0.97.61:40641)
0619 00:58:40.692420 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.692424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.692439 (+ 15us) server_negotiation.cc:407] Connection header received
0619 00:58:40.693486 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.693490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.693552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.693622 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.695455 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.696156 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.697434 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.698063 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.698192 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.702786 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.702796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.702804 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.702851 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.706109 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.706887 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.706895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.706899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.707039 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.707426 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.707432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.707435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.707714 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.708007 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.708286 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.708533 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:58:40.710126 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: Bootstrap starting.
I20260619 00:58:40.713050 23273 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 4
I20260619 00:58:40.714051 23422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:40.714339 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:58:40.715370 23422 log.cc:824] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:40.719108 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: No bootstrap required, opened a new log
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:58:40.717439 23248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2cbbbf6da815447d823dafb5d330e0b4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51598
I20260619 00:58:40.729403 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.713741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.0.97.60:33837)
0619 00:58:40.714445 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.714452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.714473 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:58:40.714537 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.714547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.714624 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.714721 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.715645 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.716629 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.718171 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.718706 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.718848 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.722484 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.722491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.722497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.722536 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.726352 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.727166 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.727172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.727176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.727334 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.727771 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.727776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.727780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.728173 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.728338 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.728771 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.729027 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531}
I20260619 00:58:40.729398 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.713696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40272)
0619 00:58:40.713920 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.713944 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.714067 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.714783 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.714787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.715285 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.715502 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.715514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.716800 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.716833 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.717997 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.718011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.718864 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.718873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.719090 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.719811 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.719830 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.722269 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.726504 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.726511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.726517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.727034 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.727455 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.727460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.727464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.727648 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.728299 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.728304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.728475 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.728623 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.728996 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 00:58:40.730381 23422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.730857 23422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75627595593f47158c27863188923a37, State: Initialized, Role: FOLLOWER
I20260619 00:58:40.731520 23422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.732650 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.733137 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:40.733964 23422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:40.735608 23273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: Bootstrap starting.
I20260619 00:58:40.740367 23273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:40.741680 23273 log.cc:824] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: Log is configured to *not* fsync() on all Append() calls
W20260619 00:58:40.745080 23440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:58:40.745205 23440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:58:40.742861 23355 master_runner.cc:186] Error getting master registration for 127.0.97.62:34697: OK, instance_id { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" instance_seqno: 1781830719566717 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:58:40.751166 23273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: No bootstrap required, opened a new log
I20260619 00:58:40.751204 23355 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 00:58:40.759258 23273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.759742 23273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbbbf6da815447d823dafb5d330e0b4, State: Initialized, Role: FOLLOWER
I20260619 00:58:40.760291 23273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.761415 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.761988 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:40.762960 23273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:40.763099 23248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 00:58:40.763337 23248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:40.767956 23248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.771358 23248 leader_election.cc:288] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [CANDIDATE]: Term 1 election: Requested vote from peers cf54dde6795c46b087e951f0f6979270 (127.0.97.61:40641), 75627595593f47158c27863188923a37 (127.0.97.60:33837)
I20260619 00:58:40.774735 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cbbbf6da815447d823dafb5d330e0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "75627595593f47158c27863188923a37"
I20260619 00:58:40.775398 23397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:40.779455 23397 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbbbf6da815447d823dafb5d330e0b4 in term 1.
I20260619 00:58:40.780398 23222 leader_election.cc:302] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbbbf6da815447d823dafb5d330e0b4, 75627595593f47158c27863188923a37; no voters:
I20260619 00:58:40.781153 23443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:58:40.781936 23443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2cbbbf6da815447d823dafb5d330e0b4, State: Running, Role: LEADER
I20260619 00:58:40.782675 23443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
W20260619 00:58:40.805017 23227 tablet.cc:2369] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:58:40.814291 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbbbf6da815447d823dafb5d330e0b4. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.814790 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:40.824469 23459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:58:40.826902 23347 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } attempt: 5
I20260619 00:58:40.841224 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.795621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.97.60:33837)
0619 00:58:40.795847 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.795855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.803722 (+ 7867us) server_negotiation.cc:407] Connection header received
0619 00:58:40.804981 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.804990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.805091 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.805171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.809880 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.810990 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.825037 (+ 14047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.825778 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.825895 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.832941 (+ 7046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.832952 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.832960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.833022 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.837282 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.838463 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.838471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.838478 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.838684 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.839242 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.839248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.839251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.839648 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.839911 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.840532 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.840830 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 00:58:40 dist-test-slave-r7bh krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 00:58:40.867714 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.795360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.0.97.62:34697)
0619 00:58:40.795614 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.795624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.809564 (+ 13940us) server_negotiation.cc:407] Connection header received
0619 00:58:40.810068 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.810074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.810158 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.810230 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.815198 (+ 4968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.816778 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.822413 (+ 5635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.823403 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.823546 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.843384 (+ 19838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.843396 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.843404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.843494 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.848498 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.853455 (+ 4957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.853498 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.853507 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.854008 (+ 501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.856567 (+ 2559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.856579 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.856585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.857737 (+ 1152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.862282 (+ 4545us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.862445 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.863392 (+ 947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 00:58:40.870518 23459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:58:40.873057 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.804218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.97.61:40641)
0619 00:58:40.804459 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.804469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.814900 (+ 10431us) server_negotiation.cc:407] Connection header received
0619 00:58:40.815593 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.815602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.815721 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.815815 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.827634 (+ 11819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.829142 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.838541 (+ 9399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.839549 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.839693 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.862167 (+ 22474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.862178 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.862187 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.862253 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.867562 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.869622 (+ 2060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.869632 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.869638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.869867 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.870436 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.870444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.870450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.870942 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.871273 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.872091 (+ 818us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.872424 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 00:58:40.889292 23376 tablet.cc:2369] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:58:40.893118 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.827553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33837 (local address 127.0.0.1:40290)
0619 00:58:40.827935 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 00:58:40.827959 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 00:58:40.828078 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:58:40.841906 (+ 13828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:58:40.841913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:58:40.842436 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:58:40.842719 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.842738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.845345 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.845354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.847392 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:58:40.847416 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.848789 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.848798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:58:40.849134 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.849839 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:58:40.849871 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:58:40.853919 (+ 4048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:58:40.889729 (+ 35810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.889736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.889741 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.890393 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.890935 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:58:40.890943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:58:40.890948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:58:40.891126 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:58:40.891932 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:58:40.891947 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:58:40.892135 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 00:58:40.892352 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 00:58:40.892685 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260619 00:58:40.894620 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:40.828314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.0.97.60:33837)
0619 00:58:40.841531 (+ 13217us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:40.841539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:40.841562 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:58:40.841624 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:40.841632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:40.841732 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:40.841858 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:40.843632 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.845100 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.847656 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:40.848565 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:40.849564 (+ 999us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:40.854381 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:58:40.854392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:58:40.854401 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:58:40.854449 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:58:40.889432 (+ 34983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.890549 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.890561 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.890564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.890764 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:58:40.891277 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:58:40.891284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:58:40.891293 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:58:40.891687 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:58:40.891846 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:40.893881 (+ 2035us) server_negotiation.cc:299] Negotiation successful
0619 00:58:40.894176 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13079,"thread_start_us":34751,"threads_started":1}
I20260619 00:58:40.880764 23206 master_runner.cc:186] Error getting master registration for 127.0.97.61:40641: OK, instance_id { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" instance_seqno: 1781830720095446 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 00:58:40.901049 23347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: Bootstrap starting.
I20260619 00:58:40.902318 23206 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 00:58:40.909235 23347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: Neither blocks nor log segments found. Creating new log.
I20260619 00:58:40.910653 23397 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [term 1 FOLLOWER]: Refusing update from remote peer 2cbbbf6da815447d823dafb5d330e0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:58:40.910988 23347 log.cc:824] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: Log is configured to *not* fsync() on all Append() calls
I20260619 00:58:40.911882 23452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:58:40.928471 23347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: No bootstrap required, opened a new log
I20260619 00:58:40.941454 23347 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.942111 23347 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf54dde6795c46b087e951f0f6979270, State: Initialized, Role: FOLLOWER
I20260619 00:58:40.944694 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbbbf6da815447d823dafb5d330e0b4. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.945685 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:40.946062 23347 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } }
I20260619 00:58:40.947474 23467 mvcc.cc:205] Tried to move back new op lower bound from 7298378632838344704 to 7298378632332791808. Current Snapshot: MvccSnapshot[applied={T|T < 7298378632838344704}]
I20260619 00:58:40.958524 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.958949 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:40.961616 23347 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:58:40.962110 23321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:58:40.964949 23322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cbbbf6da815447d823dafb5d330e0b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf54dde6795c46b087e951f0f6979270"
I20260619 00:58:40.967260 23321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [term 1 FOLLOWER]: Refusing update from remote peer 2cbbbf6da815447d823dafb5d330e0b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:58:40.967226 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.967528 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:40.969115 23471 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:40.969635 23471 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:40.971176 23322 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbbbf6da815447d823dafb5d330e0b4 in term 1.
I20260619 00:58:40.973328 23452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:58:40.980193 23459 catalog_manager.cc:1347] Generated new cluster ID: 76f581730b064757b8d8cd1b78986e13
I20260619 00:58:40.980334 23459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:58:41.020701 23471 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbbbf6da815447d823dafb5d330e0b4. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:41.021091 23471 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:41.022526 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:41.023082 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 00:58:41.025601 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbbbf6da815447d823dafb5d330e0b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbbbf6da815447d823dafb5d330e0b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34697 } } peers { permanent_uuid: "cf54dde6795c46b087e951f0f6979270" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40641 } } peers { permanent_uuid: "75627595593f47158c27863188923a37" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33837 } } }
I20260619 00:58:41.025887 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4 [sys.catalog]: This master's current role is: LEADER
I20260619 00:58:41.035939 23459 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:58:41.038149 23459 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:58:41.058746 23459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cbbbf6da815447d823dafb5d330e0b4: Generated new TSK 0
I20260619 00:58:41.060459 23459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:58:41.064158 23489 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 00:58:41.158993 23425 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.148583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.97.60:33837)
0619 00:58:41.148739 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.148745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.155635 (+ 6890us) server_negotiation.cc:407] Connection header received
0619 00:58:41.157293 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.157301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.157402 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.157478 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.158699 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40302: BlockingRecv error: recv got EOF from 127.0.0.1:40302 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 00:58:41.163286 23426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.148150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.0.97.62:34697)
0619 00:58:41.148366 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.148373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.160855 (+ 12482us) server_negotiation.cc:407] Connection header received
0619 00:58:41.161507 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.161514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.161631 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.161706 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.163026 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51622: BlockingRecv error: recv got EOF from 127.0.0.1:51622 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 00:58:41.163276 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.156354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.97.61:40641)
0619 00:58:41.156558 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.156564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.161435 (+ 4871us) server_negotiation.cc:407] Connection header received
0619 00:58:41.161690 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.161695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.161786 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.161868 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.163026 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50358: BlockingRecv error: recv got EOF from 127.0.0.1:50358 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 00:58:41.170234 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.168041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.0.97.61:40641)
0619 00:58:41.168198 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.168202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.168753 (+ 551us) server_negotiation.cc:407] Connection header received
0619 00:58:41.169044 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.169048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.169155 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.169270 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.169985 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50362: BlockingRecv error: recv got EOF from 127.0.0.1:50362 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 00:58:41.176019 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.174025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.97.61:40641)
0619 00:58:41.174240 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.174249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.174721 (+ 472us) server_negotiation.cc:407] Connection header received
0619 00:58:41.174922 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.174930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.175010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.175115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.175811 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50370: BlockingRecv error: recv got EOF from 127.0.0.1:50370 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 00:58:41.182641 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.180638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.0.97.61:40641)
0619 00:58:41.180870 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.180889 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.181352 (+ 463us) server_negotiation.cc:407] Connection header received
0619 00:58:41.181553 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.181559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.181636 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.181740 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.182411 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50376: BlockingRecv error: recv got EOF from 127.0.0.1:50376 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 00:58:41.191686 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.189507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.0.97.61:40641)
0619 00:58:41.189785 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.189794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.190332 (+ 538us) server_negotiation.cc:407] Connection header received
0619 00:58:41.190544 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.190552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.190630 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.190739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.191459 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50380: BlockingRecv error: recv got EOF from 127.0.0.1:50380 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260619 00:58:41.205184 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.202903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.97.61:40641)
0619 00:58:41.203162 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.203172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.203724 (+ 552us) server_negotiation.cc:407] Connection header received
0619 00:58:41.203946 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.203952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.204040 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.204156 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.204924 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50394: BlockingRecv error: recv got EOF from 127.0.0.1:50394 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 00:58:41.227618 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.225270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.0.97.61:40641)
0619 00:58:41.225505 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.225514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.226055 (+ 541us) server_negotiation.cc:407] Connection header received
0619 00:58:41.226281 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.226293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.226369 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.226500 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.227354 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50406: BlockingRecv error: recv got EOF from 127.0.0.1:50406 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 00:58:41.266072 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.263700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.0.97.61:40641)
0619 00:58:41.263916 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.263926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.264502 (+ 576us) server_negotiation.cc:407] Connection header received
0619 00:58:41.264746 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.264751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.264882 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.265011 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.265789 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50418: BlockingRecv error: recv got EOF from 127.0.0.1:50418 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 00:58:41.337725 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.335075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.0.97.61:40641)
0619 00:58:41.335327 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.335337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.336234 (+ 897us) server_negotiation.cc:407] Connection header received
0619 00:58:41.336487 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.336491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.336566 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.336683 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.337490 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50424: BlockingRecv error: recv got EOF from 127.0.0.1:50424 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 00:58:41.471701 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.469653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.97.61:40641)
0619 00:58:41.469869 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.469875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.470278 (+ 403us) server_negotiation.cc:407] Connection header received
0619 00:58:41.470448 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.470453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.470559 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.470694 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.471429 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53902: BlockingRecv error: recv got EOF from 127.0.0.1:53902 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 00:58:41.733179 23352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:58:41.731450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.97.61:40641)
0619 00:58:41.731710 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:41.731715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:41.731819 (+ 104us) server_negotiation.cc:407] Connection header received
0619 00:58:41.732000 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:41.732004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:41.732076 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:41.732167 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:58:41.732973 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53914: BlockingRecv error: recv got EOF from 127.0.0.1:53914 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 00:58:41.747306 23440 catalog_manager.cc:1259] Loaded cluster ID: 76f581730b064757b8d8cd1b78986e13
I20260619 00:58:41.747411 23440 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: loading cluster ID for follower catalog manager: success
I20260619 00:58:41.750114 23440 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: acquiring CA information for follower catalog manager: success
I20260619 00:58:41.752183 23440 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75627595593f47158c27863188923a37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:58:42.068485 23489 catalog_manager.cc:1259] Loaded cluster ID: 76f581730b064757b8d8cd1b78986e13
I20260619 00:58:42.068607 23489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: loading cluster ID for follower catalog manager: success
I20260619 00:58:42.071458 23489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: acquiring CA information for follower catalog manager: success
I20260619 00:58:42.073478 23489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf54dde6795c46b087e951f0f6979270: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 00:58:42.258911 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:42.251727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.0.97.61:40641)
0619 00:58:42.252407 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:42.252413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:42.252766 (+ 353us) server_negotiation.cc:407] Connection header received
0619 00:58:42.253011 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:42.253015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:42.253152 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:42.253264 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:42.253812 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.254599 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.256461 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.256933 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.257022 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:42.257608 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:42.258190 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:42.258309 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:42.258458 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 00:58:42.258523 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":335,"threads_started":1}
I20260619 00:58:42.274457 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:42.266746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.97.60:33837)
0619 00:58:42.267315 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:42.267323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:42.267664 (+ 341us) server_negotiation.cc:407] Connection header received
0619 00:58:42.267964 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:42.267969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:42.268143 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:42.268317 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:42.268892 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.269612 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.271597 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.272366 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.272516 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:42.273248 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:42.273796 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:42.273974 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:42.274042 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 00:58:42.274132 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
I20260619 00:58:42.301326 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:42.281141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.97.60:33837)
0619 00:58:42.281321 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:42.281327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:42.293626 (+ 12299us) server_negotiation.cc:407] Connection header received
0619 00:58:42.294025 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:42.294035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:42.294172 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:42.294257 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:42.295141 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.296541 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.299029 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.299667 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.299776 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:42.300434 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:42.300568 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:42.300632 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:42.300922 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 00:58:42.301012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:58:42.304927 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:42.293854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.0.97.61:40641)
0619 00:58:42.294132 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:42.294141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:42.297805 (+ 3664us) server_negotiation.cc:407] Connection header received
0619 00:58:42.297968 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:42.297976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:42.298081 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:42.298169 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:42.299085 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.300223 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.302718 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.303364 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.303460 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:42.304064 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:42.304206 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:42.304311 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:42.304478 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 00:58:42.304561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 00:58:42.305550 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:58:42.280659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.0.97.62:34697)
0619 00:58:42.297415 (+ 16756us) server_negotiation.cc:206] Beginning negotiation
0619 00:58:42.297422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:58:42.297442 (+ 20us) server_negotiation.cc:407] Connection header received
0619 00:58:42.297580 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:58:42.297584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:58:42.297784 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:58:42.298889 (+ 1105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:58:42.299080 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.300179 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.302582 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:58:42.303236 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:58:42.303848 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:58:42.304046 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:58:42.304473 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:58:42.304982 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0619 00:58:42.305037 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 00:58:42.305252 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16594,"thread_start_us":17320,"threads_started":1}
I20260619 00:58:42.310726 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 23206
I20260619 00:58:42.311803 23268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:42.455276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 23206
W20260619 00:58:42.477162 23297 connection.cc:535] server connection from 127.0.0.1:50334 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 00:58:42.477759 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 23280
I20260619 00:58:42.478894 23342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:42.589526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 23280
I20260619 00:58:42.610390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6V91bC/build/asan/bin/kudu pid 23355
I20260619 00:58:42.611799 23417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:58:42.721194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6V91bC/build/asan/bin/kudu with pid 23355
2026-06-19T00:58:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4993 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9438 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (371267 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST